Tracing network packets by a cluster of network controllers

ABSTRACT

Some embodiments provide a method for a first network controller that manages a set of logical forwarding elements implemented in several managed forwarding elements. The method receives a request to trace a specified packet having a particular source on a logical forwarding element. The method generates the packet according to the packet specification. The generated packet includes an indicator that the packet is for a trace operation. The method sends the packet to a second network controller that manages a managed forwarding element associated with the particular source. The method receives a first set of messages regarding operations performed on the packet from a set of network controllers that receives a second set of messages regarding operations performed on the packet from a set of managed forwarding elements that process the packet.

BACKGROUND

Typical physical networks include many routers and switches throughwhich packets travel. In some cases for such networks, an administratormay identify that packets are not being delivered correctly, andtherefore may wish to troubleshoot the network. In order to do so, somephysical networks have the ability to perform a route trace that showshow far a packet travels. In physical networks, the administrator canoften log into the routers and troubleshoot the routing table.

However, in at least some virtualized networks that operate manyseparate logical networks over the physical network, packets aretypically sent across the physical network in tunnels between managedforwarding elements. These tunneled packets are passed through theunmanaged physical forwarding elements (e.g., standard routers) withminimal processing, and inspecting the routers will not provide anadministrator with useful information. Thus, methods for troubleshootingvirtualized networks are needed.

BRIEF SUMMARY

Some embodiments provide a cluster of network controllers for managingforwarding elements that performs novel packet tracing operations. Insome embodiments, the cluster of network controllers includes (i) afirst set of network controllers that define and manage logical networksimplemented by the managed forwarding elements and (ii) a second set ofnetwork controllers that are responsible for managing the operations ofthe managed forwarding elements.

In some embodiments, a logical network controller receives a request totrace a specified packet having a particular source on a logicalforwarding element. In some embodiments, the packet specifies a sourceaddress and a destination address that are associated with one or morelogical forwarding elements (e.g., on one or more logical layer 2 (L2)domains, which may be connected by a logical layer 3 (L3) router). Inresponse to the request, the logical network controller generates atraceable packet from the source address to the destination address. Thelogical network controller identifies a physical network controller thatmanages a managed forwarding element (MFE) to which an entity (e.g., avirtual machine (VM) located at the source address connects). Thelogical network controller passes the packet to the identified physicalnetwork controller.

The physical network controller inserts the traceable packet into theMFE associated with the source of the packet. After inserting thetraceable packet, the physical network controller receives a set ofobservations from the MFE that indicate certain operations performed onthe packet. As the packet traverses a path to its destination, thepacket is processed and forwarded by a set of MFEs that are managed by aset of physical network controllers in the cluster. Each of thesephysical network controllers receives a set of observations from one ormore MFEs that the physical network controller manages. In someembodiments, the physical network controllers analyze the receivedobservations and send the analyses to the logical network controllerthat generated the traceable packet. Based on the messages and/oranalyses from the physical network controllers, the logical networkcontroller generates a report regarding the packet tracing operation,for delivery to a user that requested the trace.

In some embodiments, the request is received at the logical networkcontroller through an application programming interface (API) command. Auser (e.g., a network administrator), using one of a variety of userinterface tools, designs a packet to be traced through the physicalnetwork managed by the controller. In addition to the source anddestination addresses, the user may specify whether to trace a broadcastpacket (i.e., instead of a specific destination address), a payload forthe packet, the packet size, or other information.

The logical network controller generates the packet, and in someembodiments inserts an indicator into a particular location in thepacket that specifies the packet as a traced packet. For instance, someembodiments use a single bit at a specific location in the packet header(e.g., a logical VLAN field) that flags the packet as being used for atrace operation. The logical network controller then sends the packet tothe physical controller that manages a particular MFE associated withthe source of the packet (e.g., the MFE to which the entity having thesource address connects). The physical controller inserts this packetinto the appropriate MFE. In some embodiments, the MFE is a softwareforwarding element that operates in a physical host machine along withthe VM associated with the source address. After sending the packet tothe appropriate physical network controller, the logical networkcontroller then awaits the receipt of analyses of the observations fromthe physical network controllers that manage the MFEs through which thepacket passes.

In some embodiments, each MFEs through which the packet passes performsa set of logical forwarding operations and a set of physical forwardingoperations on the packet, as though the packet was an unmarked packetoriginating from a VM or other source on the network. By performinglogical forwarding operations, the MFEs advance the packet through thelogical networks towards the destination. The MFEs perform the physicalforwarding operations to advance the packet through the physicalnetworks that implement the logical networks. In some embodiments, theMFEs send an observation after performing a physical forwardingoperation only. In other embodiments, the MFEs send an observation afterperforming a logical forwarding operation as well as after performing aphysical forwarding operation.

In some embodiments, the logical forwarding operations that the MFEsperform include logical L2 ingress ACL operations, logical L2 forwardingoperations, and logical L2 egress ACL operations. The logical L2 ingressACL operations determine whether a logical forwarding element for alogical L2 domain (i.e., a logical switch) should accept an incomingpacket. The logical L2 forwarding operations include operations thatforward packets from one logical port to another logical port of alogical forwarding element for a logical L2 domain. These operationsdetermine a destination logical port on the logical switch based ondestination address information stored in the packet in someembodiments. The logical L2 egress ACL operations determine whether alogical forwarding element for a logical L2 domain should allow a packetto exit the logical forwarding element.

The logical forwarding operations of some embodiments may additionallyinclude logical L3 ingress ACL operations, logical L3 forwardingoperations, and logical L3 egress ACL operations. The logical L3 ingressACL operations determine whether a logical forwarding element forrouting packets at L3 (i.e., a logical router) should accept an incomingpacket. The logical L3 forwarding operations include operations forrouting packets from one logical L2 domain to another logical L2 domain,or to an external address. The logical L3 egress ACL operationsdetermine whether a logical forwarding element for routing packets at L3should allow a packet to exit the logical forwarding element.

One MFE may process a packet through several stages and thereforeperform several of the above logical operations, as well as physicaloperations such as sending the packet out through a tunnel. In someembodiments, while processing a packet through several stages, the MFEsstore a register bit indicating that the packet is marked for a traceoperation. The MFE looks up this register bit while processing thepacket in order to determine that observation messages should be sent tothe controller cluster.

In order to send these observation messages, the forwarding tables ofthe MFEs of some embodiments contain entries that specify when theobservation messages should be sent. In some embodiments, these messagescontain (i) the packet being processed by the MFE as received and (ii)the contents of the registers for the packets, from which the physicalcontrollers can identify the relevant data. The forwarding table entryfor sending the observation messages, in some embodiments, specifies theMFE to copy certain data to the register and then send the registercontents to the physical controller that manages this MFE.

In some embodiments, each physical network controller that receives theobservations analyzes the received observations and sends the analysesto the logical network controller. Once the logical network controllerreceives the analyses of the observations (or once a timeout set for thetrace operation is reached), the logical network controller of someembodiments generates a report and delivers it to the requesting user.In some embodiments, this report indicates whether the packet wasdelivered and provides information about each of the receivedobservations.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 conceptually illustrates a logical forwarding element 100implemented in a physical network.

FIG. 2 conceptually illustrates an example for a traced packet that issent directly from a MFE to a destination MFE.

FIG. 3 conceptually illustrates another example for a traced packet thatis sent directly from a MFE to the destination MFE.

FIG. 4 conceptually illustrates that a MFE sends multiple observationsto a controller cluster while processing a trace packet.

FIG. 5 conceptually illustrates a network controller with packetgeneration and tracing capability, as well as a MFE that analyzes andforwards packets and includes the capability to send observations fortraceable packets to the controller.

FIG. 6 conceptually illustrates a process performed by the networkcontroller of some embodiments in order to execute a packet tracingoperation.

FIG. 7 conceptually illustrates the processing performed by a MFE thatreceives a packet from a network controller, processes the packet, sendsthe packet out over a tunnel, and sends an observation to the networkcontroller.

FIG. 8 conceptually illustrates the processing performed by a MFE thatreceives a packet from another MFE through a tunnel, processes thepacket, identifies that the packet should be delivered to a logicalport, and sends an observation to the network controller.

FIG. 9 conceptually illustrates an example for a traced packet that issent through a pool node between the source MFE and destination MFE.

FIG. 10 conceptually illustrates a third example for a traced broadcastpacket.

FIG. 11 conceptually illustrates a logical network controller, aphysical network controller, and a MFE.

FIG. 12 conceptually illustrates a process performed by a logicalnetwork controller of some embodiments in order to execute a packettracing operation.

FIG. 13 conceptually illustrates a process performed by a physicalnetwork controller of some embodiments in order to execute a packettracing operation.

FIG. 14 conceptually illustrates an example for a traced packet that isgenerated by a logical controller and injected to a MFE by a physicalcontroller.

FIG. 15 conceptually illustrates another example for a traced packetthat is generated by a logical controller and injected to a MFE by aphysical controller.

FIG. 16 conceptually illustrates an example for a trace packet that isgenerated by a logical controller.

FIG. 17 conceptually illustrates an example for two trace packetsoriginating from two different logical controllers.

FIG. 18 conceptually illustrates packet processing performed by a MFE.

FIG. 19 conceptually illustrates a processing performed by a MFE thatreceives a packet from another MFE through a tunnel.

FIG. 20 conceptually illustrates an example of a MFE that sends multipleobservations to a controller cluster while processing a trace packet.

FIG. 21 conceptually illustrates an example of a processing pipelinethat a MFE performs.

FIG. 22 conceptually illustrates an example of performing logicalforwarding operations in a distributed manner by MFEs.

FIG. 23 conceptually illustrates an electronic system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

Some embodiments provide a cluster of network controllers for managingforwarding elements that performs novel packet tracing operations. Insome embodiments, the cluster of network controllers includes (i) afirst set of network controllers that define and manage logical networksimplemented by the managed forwarding elements and (ii) a second set ofnetwork controllers that are responsible for managing the operations ofthe managed forwarding elements.

In some embodiments, a logical network controller (also referred to as alogical controller) receives a request to trace a specified packethaving a particular source on a logical forwarding element. In someembodiments, the packet specifies a source address and a destinationaddress that are associated with one or more logical forwarding elements(e.g., on one or more logical layer 2 (L2) domains, which may beconnected by a logical layer 3 (L3) router). In response to the request,the logical controller generates a traceable packet from the sourceaddress to the destination address. The logical controller identifies aphysical network controller (also referred to as a physical controller)that manages a managed forwarding element (MFE) to which an entity(e.g., a virtual machine (VM) located at the source address connects).The logical controller passes the packet to the identified physicalcontroller.

The physical controller inserts the traceable packet into the MFEassociated with the source of the packet. After inserting the traceablepacket, the physical controller receives a set of observations from theMFE that indicate certain operations performed on the packet. As thepacket traverses a path to its destination, the packet is processed andforwarded by a set of MFEs that are managed by a set of physicalcontrollers in the cluster. Each of these physical controllers receivesa set of observations from one or more MFEs that the physical controllermanages. In some embodiments, the physical controllers analyze thereceived observations and send the analyses to the logical controllerthat generated the traceable packet. Based on the messages and/oranalyses from the physical controllers, the logical controller generatesa report regarding the packet tracing operation, for delivery to a userthat requested the trace.

In some embodiments, the cluster of network controllers (also referredto as controller cluster) manages one or more logical forwardingelements implemented across physical forwarding elements in adistributed, virtualized environment. That is, rather than using asingle physical switch to implement a logical forwarding element, theforwarding responsibilities are spread across MFEs distributedthroughout the network. For instance, some embodiments include switchingsoftware within physical host machines (e.g., running on top of orwithin a hypervisor on the host). This switching software (e.g., openvirtual switch (“OVS”)) implements the logical forwarding elements ofsome embodiments.

FIG. 1 conceptually illustrates a logical forwarding element (e.g., alogical switch) 100 implemented in a physical network 150. As shown, thelogical forwarding element 100 connects six virtual machines 105-130.Each of these virtual machines 105-130 connects to a logical port of thelogical forwarding element 100. In some embodiments, a user (e.g.,network administrator) defines the logical forwarding element, which maybe part of a larger logical network. For instance, the logicalforwarding element is a logical L2 switch that may include a logicalport that connects to an external gateway (e.g., to an externalnetwork), to a logical L3 router (which may also connect to otherlogical L2 switches), etc.

In some embodiments, the user defines the logical forwarding element 130through an API of a controller cluster, which translates the userdefinition into a logical control plane definition of the logicalforwarding element 130. The controller cluster then converts thislogical control plane definition into a logical forwarding planespecification of the logical forwarding element. The logical forwardingplane specification, in some embodiments, includes logical forwardingtable entries (logical flow entries) that specify rules for forwardingpackets to logical ports of the logical forwarding element. Forinstance, the logical control plane of some embodiments includesbindings between media access control (MAC) addresses of VMs and logicalports, and the logical forwarding plane specifies flow entries forforwarding packets to the logical ports based on matches of the MACaddresses.

In addition, the controller cluster of some embodiments converts thelogical forwarding plane data into physical control plane data thatspecifies rules for the MFEs to follow in order to implement the logicalforwarding element. This physical control plane data includes matchesover the logical forwarding element itself (e.g., based on the source ofthe packet), as well as entries for placing packets into tunnels fromone MFE to another (and receiving packets from these tunnels). Theserules, in some embodiments, incorporate data from the MFEs, such asphysical ports and tunnel IP address information. The controller clusterthen pushes this physical control plane data down to the MFEs.

In some embodiments, the different network controllers (i.e., logicaland physical controllers) in the controller cluster take part indifferent portions of the translation from logical control planedefinition into the logical forwarding plane definition and then intothe physical control plane data. The controller cluster and generationof flow entries of some embodiments is described in greater detail inthe U.S. Publication 2013/0103817, which is incorporated herein byreference.

The controller cluster, as mentioned, pushes these flow entries toseveral MFEs in some embodiments, such that the logical forwardingelement (and/or other logical forwarding elements, such as logicalrouters) is implemented in a distributed, virtualized fashion. Thephysical network 150 of FIG. 1 illustrates that the six VMs 105-130 arehosted on four different host machines 155-170. Some embodiments mayonly host one VM from a particular logical network on a single machine,while other embodiments may put multiple VMs from a logical network onthe same machine, as in this case with the hosts 155 and 170. As shown,in the virtualized environment, each of these hosts 155-170 also hostsadditional virtual machines beyond those connected to the logicalforwarding element 100. That is, many tenants may share the use of thephysical network 150, and in fact may share use of a single physicalhost.

Operating on each host (e.g., within the hypervisor on the host) is aMFE. The MFE, in some embodiments, is a software forwarding element towhich the network controller connects and pushes down flow entries forvarious logical forwarding elements. In this case, because VMs from thelogical forwarding element 100 are located on each of the fourillustrated hosts 155-170, the MFE in each of these hosts implements thelogical forwarding element 100. That is, each of the illustrated MFEshas flow entries in its forwarding tables for logically forwardingpackets to the logical ports associated with the different VMs 105-130.

In addition to the host machines 155-170, the physical network 150further includes a pool node 175, also referred to as a service node insome embodiments. The pool node 175 connects to each of the hosts withinthe network, and serves to forward packets between edge MFEs (thoselocated in the hosts, at the edge of the network) that do not havedirect connections. As shown, the first MFE in host 155 and second MFEin host 160 have a tunnel connection between them, as do the third MFEin host 165 and the fourth MFE in host 170. However, the first MFE doesnot have a direct tunnel connection to the third or fourth MFE, nor doesthe second MFE. Instead, each of the four MFEs have tunnels defined toports of the pool node 175. In some embodiments, packets sent along eachof these tunnels pass through one or more unmanaged forwarding elements(e.g., standard, dedicated routers) that do not receive flow entriesfrom the network controller and pass along the packets with only minimalprocessing.

In addition, the pool node 175 may include connections to a remotedomain in some embodiments (e.g., a separate domain that implements adifferent logical L2 switch). The logical forwarding element 300 couldinclude a seventh port for connecting to this remote domain, and packetssent from one of the VMs 105-130 to the remote domain would be forwardedby the pool node through a physical connection to the remote domain.

In some embodiments, tunnels provided by tunneling protocols (e.g.,control and provisioning of wireless access points (CAPWAP), genericroute encapsulation (GRE), GRE Internet Protocol Security (IPsec), etc.)may be used to facilitate the implementation of the logical forwardingelement 101 the four MFEs. By tunneling, a packet is transmitted throughthe switches and routers as a payload of another packet. That is, atunneled packet does not have to expose its addresses (e.g., source anddestination MAC addresses) as the packet is forwarded based on theaddresses included in the header of the outer packet that isencapsulating the tunneled packet. Tunneling, therefore, allowsseparation of logical address space from the physical address space as atunneled packet can have addresses meaningful in the logical addressspace while the outer packet is forwarded/routed based on the addressesin the physical address space. In this manner, the tunnels may be viewedas the “logical wires” that connect MFEs in the network in order toimplement the logical forwarding element.

Within the above-described environment, in which the controller clusterconnects to each of the MFEs (including the pool node), in someembodiments the controller cluster receives a request through an APIcommand. A user (e.g., a network administrator), using one of a varietyof user interface tools, designs a packet to be traced through thephysical network managed by the controller cluster. In addition to thesource and destination addresses, the user may specify whether to tracea broadcast packet (i.e., instead of a specific destination address), apayload for the packet, the packet size, or other information.

The controller cluster generates the packet, and in some embodimentsinserts an indicator into a particular location in the packet thatspecifies the packet as a traced packet. For instance, some embodimentsuse a single bit at a specific location in the packet header (e.g., alogical VLAN field) that flags the packet as being used for a traceoperation. The controller cluster then sends the packet to theparticular MFE associated with the source of the packet (e.g., themanaged forwarding element to which the entity having the source addressconnects). Some embodiments additionally set registers associated withthe packet at the particular managed forwarding element in order tosimulate the MFE receiving the packet through a particular physical portassociated with the source address.

In some embodiments, this MFE is a software forwarding element thatoperates in a physical host machine along with the VM associated withthe source address. The controller cluster then awaits the receipt ofobservations from the MFEs through which the packet passes.

In some embodiments, the MFEs through which the packet passes eachperforms a set of logical forwarding operations and a set of physicalforwarding operations on the packet like the forwarding element doeswhen processing an unmarked packet actually received from a VM. Byperforming logical forwarding operations, the MFEs advance the packetthrough the logical networks towards the destination. The MFEs performthe physical forwarding operations to advance the packet through thephysical networks that implement the logical networks. In someembodiments, the MFEs send an observation after performing a physicalforwarding operation only. In other embodiments, the managed forwardingelements send observations after performing a logical forwardingoperation as well as after performing a physical forwarding operation.

In some embodiments, the packet tracing operation operates with aspecified timeout after which the controller cluster assumes that noadditional observations will be delivered. Other than sending theobservations and not actually delivering the packet to a VM (or otherdestination bound to a logical port), the MFEs process the packet in thesame manner as an unmarked packet actually received from a VM. In someembodiments, while processing a packet through several stages, the MFEsstore a register bit indicating that the packet is marked for a traceoperation.

In order to send observation messages, the forwarding tables of the MFEsof some embodiments contain entries that specify when the observationmessages should be sent. In some embodiments, these messages contain (i)the packet being processed by the MFE as received and (ii) the contentsof the registers for the packets, from which the controller can identifythe relevant data. The forwarding table entry for sending theobservation messages, in some embodiments, specifies to the MFE to copycertain data to the registers and then send the register contents to thecontroller.

Once the controller cluster receives the observations (or the timeout isreached), the controller cluster of some embodiments generates a reportand delivers it to the requesting user. In some embodiments, this reportindicates whether the packet was delivered and provides informationabout each of the received observations.

In some embodiments, the controller cluster includes a single networkcontroller (or a single master controller with a backup controller) thatperforms translation of the logical control plane definition. Thissingle network controller also generates a trace packet, receivesobservations from the MFEs, and generates a report from the receivedobservations. FIG. 2 conceptually illustrates an example in which thecontroller cluster includes such a single network controller thatgenerates a traced packet and receives observations from the MFEsthrough which the traced packet passes. In this example, the tracedpacket is injected into a first MFE. The traced packet is then sentdirectly from the first MFE to a destination MFE. As shown, this exampleillustrates a controller 205, a first edge MFE 210, and a second edgeMFE 215. In addition, the figure illustrates various packets andobservations with encircled numbers that indicate an order in which thevarious data (packets, observation messages) are sent within thenetwork.

As shown by the encircled 1, initially the single controller 205receives a request (e.g., as will be described further below byreference to FIG. 5). In this case, the request specifies a sourcelogical port that maps to a VM connected to the first MFE 210 and adestination logical port that maps to a VM connected to the second MFE215. In response to the request, the controller 205 generates a packet220 and sends this to the MFE (shown by the encircled 2), along with acommand to process the packet. The header of the packet 220 specifiesthe source and destination addresses received in the request, as well asa trace indicator that marks the packet as a test packet for a traceoperation.

The first edge MFE 210 processes the packet (e.g., as will be shownfurther below in FIG. 7), and its forwarding table entries in someembodiments indicate to (i) encapsulate the packet with a logicalcontext tag that specifies a destination logical port corresponding tothe destination MAC address and (ii) further encapsulate the packetusing tunnel encapsulation with an IP address for a port of the secondMFE 215. Furthermore, the forwarding table entries also specify to sendan observation message to the controller because a register bit at theMFE has been modified to identify the packet as associated with a traceoperation. Thus, as shown by the encircled 3, the first MFE 210 sendsboth an observation 225 to the controller 205 and a packet 230 to thesecond MFE 215. In some embodiments, the first MFE 210 sends more thanone observation message to the controller 205. In these embodiments, thefirst MFE 210 sends an observation message after performing a logical orphysical forwarding operation in a set of logical and physicalforwarding operations that the MFE 210 performs on the packet 230 beforesending the packet to the second MFE 215. Sending more than oneobservation message from a MFE will be further described below byreference to FIG. 4. The packet 230 is the same as the packet 220 insome embodiments, with the additional encapsulations mentioned above.

The MFE 215 receives this packet 230, which still includes the traceindicator, and is encapsulated with the destination logical port. Thereceiving MFE 215 reads this destination logical port and identifies(via its forwarding table entries) to deliver the packet to the VMassociated with the destination logical port. However, based onadditional entries that (i) modify a register at the MFE to indicate thetrace packet and (ii) read this register when delivering to the logicalport, the MFE 215 instead drops the packet and sends an observation 235(indicated by the encircled 4) to the controller 205. In someembodiments, this is a delivery observation message that specifies theID of the MFE 215 and the port to which the packet would have beendelivered, as will be described by reference to FIG. 8.

FIG. 2 above illustrates an example of a trace operation when thecontroller cluster includes a single network controller. In contrast,FIG. 3 conceptually illustrates an example of a trace operation when thecontroller cluster includes several different network controllersforming a hierarchy of controllers. As shown, a controller cluster 305in this example includes a logical controller 310 and two physicalcontrollers 315 and 320. This figure also illustrates a first edge MFE325 and a second edge MFE 330 along with various packets andobservations with encircled numbers that indicate an order in which thepackets and observations messages sent within the network.

As shown by the encircled 1, the logical controller 310 receives arequest from the user. This request specifies a source logical port thatmaps to a VM connected to the MFE 325. In response to the request, thelogical controller 310 generates a packet 335. As shown by the encircled2, the logical controller then sends the generated packet to thephysical controller 315 because the physical controller 315 manages theMFE 325. In some embodiments, a particular physical controller thatmanages a particular MFE is the only controller other than a back-up ofthe particular physical controller that can exchange data with the MFE.The header of the packet 335 specifies the source and destinationaddresses received in the request, as well as a trace indicator thatmarks the packet as a test packet for a trace operation.

The physical controller 335 examines the packet and identifies that theMFE 325 is the MFE into which to inject the packet because the logicalport to which the source address of the packet connects is mapped to aphysical port of the MFE 325. As shown by the encircled 3, the physicalcontroller 315 sends the packet 335 to the MFE 325.

The processing of the received packet by the first edge MFE 325 issimilar to the processing of the trace packet by the first edge MFE 210in FIG. 2. That is, the first edge MFE 325 processes the packet and itsforwarding table entries indicate to (i) encapsulate the packet with alogical context tag that specifies a destination logical portcorresponding to the destination MAC address and (ii) furtherencapsulate the packet using tunnel encapsulation with an IP address fora port of the second MFE 330. Furthermore, the forwarding table entriesalso specify to send an observation message to a physical controller 315that sent the packet to the MFE because a register bit at the MFE hasbeen modified to identify the packet as associated with a traceoperation. Thus, as shown by the encircled 4, the first MFE 325 sendsboth an observation 340 to the physical controller 315 and a packet 345to the second MFE 330. In some embodiments, the MFE 325 sends more thanone observation message to the physical controller 315 before sendingthe packet to the second MFE 330. In these embodiments, the first MFE325 sends an observation message to the physical controller afterperforming a logical or physical forwarding operation in a set oflogical and physical forwarding operations that the MFE 325 performs onthe packet 335. The packet 345 is the same as the packet 335 in someembodiments, with the additional encapsulations mentioned above.

When the physical controller 315 receives the observation 340 from thefirst MFE 340, the physical controller 315 of some embodiments analyzesthe observation and sends (as shown by the encircled 5) the analysis ofthe observation to the logical controller 310. The logical controller310 receives the analysis of the observation and waits for more analysesto come from other physical controllers until a timeout is reached. Inother embodiments, the physical controller 315 relays the observationmessage to the logical controller 310 and the logical controller 310analyzes the observation.

The second MFE 330 receives the packet 345, which still includes thetrace indicator, and is encapsulated with the destination logical port.The receiving MFE 330 reads this destination logical port and identifies(via its forwarding table entries) to deliver the packet to the VMassociated with the destination logical port by performing a set oflogical and physical forwarding operations on the packet 345. However,based on additional entries that (i) modify a register at the MFE toindicate the trace packet and (ii) read this register when delivering tothe logical port, the MFE 330 instead drops the packet and sends anobservation 350 (indicated by the encircled 6) to the physicalcontroller 320. In some embodiments, the MFE 330 sends the observationto the physical controller 320 because the physical controller 320manages the MFE 330.

In some embodiments, the MFE 330 sends multiple observations to thephysical controller 320. Instead of only sending the observation fordelivery to a destination (a physical forwarding operation), the MFE 330sends an observation after performing each logical or physicalforwarding operation in a set of logical and physical forwardingoperations that the MFE 330 performs on the packet 345.

Like the physical controller 315, the physical controller 320 of someembodiments analyzes the received observation(s) and sends (as shown bythe encircled 7) the analysis of the observation to the logicalcontroller 310. In other embodiments, the physical controller 315 relaysthe observation message to the logical controller 310 and the logicalcontroller 310 analyzes the observation. The logical controller 310receives the observation(s) or the analysis of the observation(s) andwaits for more analyses/observations to come from other physicalcontrollers until a timeout is reached. Once the timeout is reached, thelogical controller 310 generates a report for the requested tracingoperation and sends (as shown by the encircled 8) the report to theuser.

It is to be noted that the order between sending (shown by the encircled5) the analysis of the observation by the physical controller 315 andthe sending (shown by the encircled 6) of the observation by the MFE 330may be arbitrary even though the encircled numbers are in sequence. Thisis because the observation 340 and the packet 345 may take differentamounts of time to reach the physical controller 315 and the MFE 335,respectively. Also, the physical controller 315 and the MFE 335 may takedifferent amounts of time to process the observation 340 and the packet345, respectively.

FIG. 4 conceptually illustrates an example of a MFE that sends multipleobservations to a controller cluster while processing a trace packet.Specifically, this figure illustrates a processing pipeline 400 that aMFE performs on a packet in order to advance the packet through alogical network implemented across a set of MFEs. The processingpipeline 400 includes four stages 410-440 for processing a packet. Insome embodiments, each MFE through which the packet passes performs theprocessing pipeline 400 (or a portion of the processing pipeline) whenthe MFE receives the packet. This figure also illustrates a controllercluster 445, a MFE 450, and various packets and observations withencircled numbers that indicate an order in which the packets andobservations messages sent.

As shown by the encircled 1, the controller cluster 405 receives arequest. In this case, the request specifies a source logical port thatmaps to a VM connected to the MFE 450 and a destination logical portthat maps to a VM connected to another MFE. In response to the request,the controller 405 generates a packet 455 and sends (shown by theencircled 2) this to the MFE 450. The header of the packet 455 specifiesthe source and destination addresses received in the request, as well asa trace indicator that marks the packet as a test packet for a traceoperation.

In some embodiments, the header of the packet includes a set of fieldsthat contains information used for forwarding the packet through anetwork. Forwarding elements perform a set of forwarding operationsbased on the information contained in the header (and possibly registersfor the packet stored by the forwarding element) and may, in some cases,modify some or all of the header fields. As explained above, someembodiments perform forwarding operations based on flow entries in theforwarding elements' forwarding tables.

In some embodiments, the processing pipeline 400 may be implemented byflow entries in the forwarding tables of the MFEs. For instance, someflow entries are defined such that the packet is processed against theflow entries based on various fields in the packet's header, includingsource and/or destination address, a logical context tag generated by aprevious MFE, etc. In some embodiments, a controller cluster configuresthe MFEs to have such flow entries.

When the MFE 450 receives the packet 455, the MFE 450 stores a registerbit indicating that the packet is marked for a trace operation. The MFE450 then performs a logical context mapping on the packet to determinethe logical context of the packet in the first stage 410 of theprocessing pipeline 400. In some embodiments, the first stage 410 isperformed when a logical forwarding element receives the packet (e.g.,the packet is initially received by the MFE 450 that implements thelogical forwarding element).

In some embodiments, a logical context represents the state of thepacket with respect to the logical forwarding element. For example, thelogical context may specify the logical forwarding element to which thepacket belongs, the logical port of the logical forwarding elementthrough which the packet was received, the logical port of the logicalforwarding element to which the packet is to be transmitted, the stageof the logical forwarding plane of the logical forwarding element thepacket is at, etc.

In some embodiments, the first MFE to receive a packet determines thelogical context by mapping a physical port of the MFE through which thepacket was received and/or a source address (e.g., MAC address) of thepacket to a logical port of a particular logical forwarding element.Other embodiments may use other fields in the header of the packet(e.g., a VLAN tag) in order to initially determine the logical contextof the packet. At subsequent MFEs, the logical context may be determinedbased on a logical context tag placed in the packet header by a previousMFE.

After determining the logical context of the packet, the MFE of someembodiments stores this information in either the packet header itselfor registers created for the packet at the MFE. For instance, the firstMFE may store the logical input port in a register, then utilize thisinformation for performing the logical forwarding operations 420,described below. After performing these operations, some embodimentsidentify a logical output port, which the MFE uses to encapsulate thepacket with a logical context tag. Later MFEs can read this logicalcontext to identify the logical output port of the packet and performany necessary operations on the packet using this information.

In the second stage 420 of the processing pipeline 400, the MFE performsa set of logical forwarding operations on the packets to determine alogical port of the logical forwarding element to which the packetshould be forwarded. In some embodiments, a packet may be sent throughmultiple connected logical forwarding elements (e.g., from one logicalL2 switch to another logical L2 switch through a logical L3 router), andthese operations may collectively be referred to as logical forwardingoperations. Thus, in some embodiments, the logical forwarding operationsmay include a logical ingress ACL operation for determining accesscontrol when the logical forwarding element receives the packet, alogical L2 operation for determining where to forward the packet througha layer 2 network, and a logical egress ACL operation for determiningaccess control before the logical forwarding element routes the packetout of the logical forwarding element. Alternatively, or in conjunctionwith the logical L2 operation, some embodiments of the logicalforwarding operations include a logical L3 operation for determiningwhere to route the packet through a layer three network.

In some embodiments, the MFE 450 sends (shown by the encircled 3) a setof observations for the set of logical forwarding operations 420. Insome embodiments, the MFE 450 is configured (by the controller cluster445) to send an observation to the controller cluster after performingeach logical forwarding operation in the set of logical forwardingoperations in the stage 420 when the packet is marked for the traceoperation and/or a register bit is set indicating that the packet is fora trace operation. Because the packet 455 is marked for the tracingoperation, the MFE 450 in this example sends an observation includingthe result of performing the logical forwarding operation. In someembodiments, the MFE 450 is configured to send observations only forsome of the logical forwarding operations performed on the markedpacket.

In some embodiments, the result of the logical forwarding operations mayinclude dropping the packet, enqueuing the packet, forwarding the packetto one or more logical egress ports of the logical forwarding element,etc. In addition, many of the operations send the packet to a dispatchport of the MFE in order to resubmit the packet to the forwarding tablesof the MFE that implement the logical forwarding elements.

Next, the third stage 430 of the processing pipeline 400 performs amapping operation on the packet. In some embodiments, the mappingoperation is a logical to physical mapping operation that maps thelogical egress port of the logical forwarding element (a result oflogical forwarding operations 420) to a physical result in the network.That is, the mapping operation determines one or more ports of one ormore MFEs that correspond to the logical egress port of the logicalforwarding element through which the packet is to be sent out. Forinstance, if the packet is a broadcast packet or a multicast packet, thethird stage 430 of some embodiments determines the ports of the MFEsthat correspond to the logical egress ports of the logical forwardingelement through which the packet is to be sent out (i.e., the logicalports to which the intended recipients of the packet are coupled). Ifthe packet is a unicast packet, the third stage 430 determines a port ofa MFE that corresponds to the logical egress port of the logicalforwarding element through which the packet is to be sent out (i.e., thelogical port to which the intended recipient of the packet is coupled).

At the fourth stage 440 of the processing pipeline 400, a set ofoperations to implement physical forwarding of the packet is performed.The set of physical operations of some embodiments includes operationsfor sending the packet to the physical port(s) that corresponds to thelogical egress port(s) of the packet, as determined in the third stage430. For example, the set of physical operations of some embodimentsdetermines one or more ports of the MFE on which the processing pipeline400 is being performed through which to send the packet out in order forthe packet to reach the physical port(s) determined in the third stage430. This way, the MFEs can route the packet along the correct path inthe network for the packet to reach the determined physical port(s) thatcorresponds to the packet destination.

In some embodiments, the MFE 450 sends (shown by the encircled 4) one ormore observation messages 465 to the controller cluster for the set ofphysical operations performed at the fourth stage 440 when the registerbit indicates that the packet is marked for the trace operation. In someembodiments, the MFE 450 is configured (by the controller cluster 445)to send observations for all of the physical forwarding operations. Inother embodiments, the MFE 450 is configured to send observations onlyfor some of the physical forwarding operations performed on the markedpacket. For instance, the MFE of some embodiments sends observationswhen either of two actions is taken by the MFE: (1) the MFE sends thepacket to another MFE via a tunnel, or (2) the MFE delivers the packetto a physical port to which the logical egress port is mapped.

As mentioned above, in some embodiments, the processing pipeline 400 isperformed by each MFE in the managed network that is used to implementthe logical network. In some embodiments, some of the MFEs perform onlya portion of the processing pipeline 400. For example, in someembodiments, the MFE that initially receives the packet may perform thefirst-fourth stages 410-440 and the remaining MFEs that subsequentlyreceive the packet only perform the first, third, and fourth stages 410,430, and 440. The processing pipeline, including logical and physicalforwarding operations, that each MFE performs to process an incomingpacket is further described in the U.S. Publications 2013/0044636 and2013/0058250, which are incorporated herein by reference.

The above description introduces the packet tracing operations of someembodiments. Several more detailed embodiments are described below.First, Section I describes the operation of a single network controllerthat performs a packet tracing operation to trace operations performedon a packet. Next, Section II describes the operation of several networkcontrollers that facilitate packet tracing together. Section III thendescribes sending observations by a MFE for logical forwardingoperations. Finally, Section IV describes an electronic system withwhich some embodiments of the invention are implemented.

I. Tracing by a Single Controller

A. Single Network Controller Operation

As indicated, in some embodiments a single controller generates atraceable packet according to a request received from a user and insertsthis packet at a particular MFE. In order to generate such a packet, thenetwork controller of some embodiments knows how the packet headersshould appear when received at the MFE from a specified source port(i.e., what data should be in the packet headers). Thus, the networkcontroller can generate a packet that mimics a packet actually receivedat the source port of the MFE.

FIG. 5 conceptually illustrates a network controller 500 with packetgeneration and tracing capability, as well as a MFE 550 that analyzesand forwards packets and includes the capability to send observationsfor traceable packets to the controller 500. As shown, the controller500 includes an input interface 505, an nLog state computation module510, a packet generator 515, an observation analyzer 520, and a MFEinterface 525. In addition, the network controller 500 includes one ormore state storage database 530, which in some embodiments stores inputand/or output of the nLog state computation module 510.

The input interface 505 of some embodiments receives input from one ormore users to define logical networks (e.g., sets of workloads (such asVMs) connected through logical switches, logical routers, middleboxes,gateways to external networks, etc.), as well as requests to tracepackets sent through a logical switch on such a logical network. Forexample, a user could define a logical switch shown in FIG. 1, describedabove. In addition, with such a logical switch defined and implementedwithin the physical network managed by the network controller 500 (whichincludes the MFE 550), the user could request a packet tracing operationfor a packet sent from one logical port of the logical switch 100 toanother logical port of the logical switch.

In some embodiments, the request received at the input interfacespecifies the logical ports in terms of source and destination MACaddresses entered (or selected) by the user. In other embodiments, theuser specifies a source logical port (e.g., in terms of a UUID used torepresent the port) at which to insert the packet and a destination MACaddress that is used in the packet itself (a source MAC address will beplaced in the packet header as well, based on the specified logicalport). In addition, the request may include additional parameters beyondthe source and destination addresses. For instance, in some embodimentsthe user specifies a payload for the packet. When no payload isspecified, the controller inserts a default payload for the packet.Users can also specify a frame size and/or ethertype for the tracedpacket. In some embodiments, the user can specify a timeout whichspecifies a duration (e.g., 100 ms, 1 s, 10 s, etc.) that the controllerwill wait for observations from the MFEs in the network.

In addition, instead of specifying a specific destination addresscorresponding to a single logical port on the logical switch, in someembodiments the user may wish to send a broadcast packet on the logicalswitch. In some embodiments, when the request does not include adestination address, or does not include a destination address thatcorresponds to a particular VM, the controller generates a traceablebroadcast packet for insertion into the network at the specified sourcelogical port. In other embodiments, the user is required to specify adestination address for each trace operation, and must affirmativelyspecify a broadcast destination in order to perform the trace on abroadcast packet.

When the input interface 505 receives a specification of a logicalnetwork, the interface of some embodiments translates this specificationinto logical control plane data that defines the logical network, andpasses this data to the nLog state computation module 510. In someembodiments, the input interface 505 reads this logical control planedata into input tables of the state computation module 510. The nLogstate computation module 510 of some embodiments includes a tablemapping engine with a set of input tables and output tables, and mapsrecords in the input tables to records in the output tables according toa set of rules. More specifically, some embodiments translate logicalcontrol plane data into logical forwarding plane data and subsequentlytranslate the logical forwarding plane data into physical control planedata that can be passed down to the MFEs that implement the logicalnetwork. The nLog state computation module 510 of some embodiments isdescribed in greater detail in U.S. Publication 2013/0058228, which isincorporated herein by reference.

In some embodiments, the physical control plane data for a particularspecified logical network includes a set of flow entries used by themanaged forwarding elements to implement the logical network. Forinstance, a flow entry for performing a logical forwarding decisionmight state that if a packet matches a specific logical switch (e.g.,based on its source MAC address or ingress port) and the destinationaddress is a particular MAC address, the packet should be forwarded to aparticular logical port of the logical switch. An example of such a flowwould state “If match L2 switch Q and destination MAC B→forward tological egress port X”. The physical control plane flow entriesadditionally include entries for identifying a physical port to which tosend the packet and placing the packet in tunnels. To continue the aboveexample, the physical control plane entries might include a flow stating“If logical egress=port X→encapsulate with remote tunnel information Zand send to physical port P”.

The above control plane entries will be matched for packets that aresent over the network as part of normal network traffic or which aregenerated by the controller for a trace operation. However, someembodiments also include flow entries for the trace operation, which arepassed to the MFEs as well. These entries basically specify that if thepacket being sent over the tunnel (or delivered to a final destination)is a traced packet, then certain data should be copied to the registersand sent as an observation message to the controller. For example, usingthe above case, a flow entry might state “If sending packet with tunnelinformation Z and register bit J==1→copy MFE ID and tunnel information Zto registers and send register data with packet to controller”.Similarly, for a delivery observation message, a flow entry might state“If sending packet to physical port R and register bit J==1→drop packet,copy MFE ID and physical port ID to register and send register data withpacket to controller”. These flow entries, like the other physicalcontrol plane data, are converted by the MFE into physical forwardingplane data in the forwarding tables of the MFE, as described below.

In some embodiments, the nLog state computation module 510 stores itsoutput state in the state storage database(s) 530. This database 530stores MAC address to logical port bindings, physical control plane dataoutput by the nLog state computation module 510, and other data in someembodiments.

The operations of the nLog state computation module 510 may take placeat a different time than the packet tracing operations. That is, in someembodiments the controller 500 initially generates (using the nLog statecomputation module 510) the flow entries for a logical network, both forthe packet tracing operation and the general logical networkfunctionality. While the flow entries may require updating as thenetwork changes (i.e., due to the user adding or removing entities fromthe network, modifications to the physical network, etc.), the packettracing operations performed by both the controller and the managedforwarding elements take place at conceptually a different time (i.e.,in response to user requests, which may occur well after the generationand distribution of the flow entries by the controller 500).

The packet generator 415 receives requests to trace packets through theinput interface 405. The packet generator 415, in some embodiments,receives the source and destination logical ports, payloadspecification, frame size, etc., and generates a packet with theappropriate headers and payload. In addition, the packet generator 515appends an indicator in the packet that specifies to a MFE processingthe packet that the packet is a traced packet. This signifies to the MFE(1) that it should send observations back to the controller when itperforms specific logical or physical forwarding operations on thepacket and (2) that it should drop the packet once it has made thedecision to deliver the packet to its intended recipient at thedestination address. In addition, some embodiments append an identifierfor the issuing controller (i.e., the controller 500) as well as a traceoperation session ID to differentiate between multiple packet tracingoperations. Some embodiments append this data at the end of the payload.When appending the data to the end of the payload, the controller ofsome embodiments updates length and checksum fields of the packet toaccommodate the appended data, so that the packet remains a valid IPpacket. After generating the packet, the controller sends the packet tothe appropriate MFE (that which connects to the source logical port)through the MFE interface 525.

The observation analyzer 520 receives observations about a traced packetsent by the MFEs to the network controller 500. In some embodiments, aMFE sends a set of observations to the controller by sending anobservation whenever it performs one forwarding operation in a set oflogical or physical forwarding operations to perform on a packet. Inaddition, when a MFE delivers a traced packet to its destination (orwould deliver the packet, if it was not a traced packet generated by thenetwork controller), the MFE sends an observation to the controller.These observations are received at the MFE interface 525 and sent to theobservation analyzer 520. The structure of these observations isdiscussed in further detail below.

The observation analyzer 520 of some embodiments performs operations todeduce certain data from the received observations. For instance, forobservations indicating that the packet was forwarded into a particulartunnel, the observation analyzer uses the mappings contained in thestate storage 530 to identify the remote IP address to which the packetwas sent. For observations indicating delivery to a logical port, theobservation includes a physical port of the MFE through which the packetwould be sent, which the observation analyzer 520 maps to the logicalport using the data in the state storage 530.

In some embodiments, the observation analyzer generates a report for thetraced packet. Some embodiments provide a summary report to the userthrough the input interface 505 that indicates whether the packetsuccessfully reached its destination(s), to which destinations it wasdelivered, packet information specified in the request(source/destination addresses, frame size, timeout duration, etc.), thenumber of observations received, the number of times the packet wasforwarded, and a time stamp that indicates when the traced packet wassent by the controller.

In addition, for each observation received, some embodiments includeadditional information for presentation to the user. This informationmay include whether the observation was sent for forwarding (into atunnel) or delivery to a destination, information about the source anddestination MFE for a particular tunnel, time duration from when thepacket was sent by the controller to when the observation was received,remote IP address (for observations indicating that the packet wasforwarded into a tunnel), and logical port to which the packet wouldhave been delivered (for observations indicating delivery). Theinformation may also include whether the logical forwarding element(e.g., a logical switch, a logical router, etc.) dropped or forwardedthe packet in the logical network, the reason for the logical forwardingelement's dropping the packet (e.g., by ingress or egress ACLs, etc.),the location of the packet within the logical network, etc.

As shown, the controller connects to one or more MFEs through its MFEinterface 525. Through this interface, the controller (i) distributesphysical control plane data to the MFEs, (ii) sends traced packets forinsertion into the physical network at a particular MFE and (iii)receives observations regarding traced packets from the MFEs in thephysical network. In some embodiments, the communications channelbetween the controller and the MFE for the purpose of the packet tracingoperations is the same channel through which the physical control planeentries are pushed down to the MFE (e.g., using a particularcommunication protocol such as OpenFlow). In some embodiments, thecommunication through this interface is a TCP connection between thecontroller and a specific control port of the MFE (the interface 555,described below). Thus, IP reachability is required between thecontroller and the control port of the MFE. Some embodiments use aspecific VLAN for control channel connectivity.

While the controller 500 connects to multiple MFEs, FIG. 5 displaysadditional detail regarding a particular one of the MFEs 550, which isillustrative of all of the MFEs in some embodiments. This MFE 550 may bean edge MFE which resides in a machine that hosts VMs, a pool node, etc.The MFE 550 includes a controller interface 555, a packet processor 560,a set of forwarding tables 565, a set of registers 570, and a set ofports 575 to other MFEs.

The controller interface 555 enables the MFE to communicate with thenetwork controller 500. Through this interface, the MFE 550 receivesphysical control plane data that it converts (e.g., using a module notshown in this figure) into physical forwarding plane data that populatesthe forwarding tables 565. In addition, through the controller interface555, the MFE 550 receives traced packets for processing and sendsobservation messages back to the controller.

The packet processor 460 receives packets (e.g., from the controllerinterface, from the ports 575) and processes the packets using theforwarding tables 565. The forwarding tables 565, in some embodiments,include entries for one or more unaffiliated logical networks that arevirtualized over the physical network. Each entry includes a conditionand a corresponding action to perform if the condition is matched, insome embodiments. For instance, a table entry might specify that if aparticular bit is set to 1 in the packet (or in a register) and a packetis being sent through a tunnel, then send a particular observationmessage to the controller. Another example of a forwarding table entryis that if a packet is received from a particular physical port, thenset a particular logical ingress port for the packet in a register.

The registers 570 are used by the packet processor 560 as a temporarystorage when processing a packet (e.g., in memory). In some embodiments,the packet processor 560 uses one or more registers per packet, suchthat numerous registers may be created at a single time when numerousdifferent packets are being processed. The temporary information mayinclude an indication that a packet is for a tracing operation, a resultof a logical or physical forwarding operation performed on the packet,logical context information that specifies information about the logicalforwarding element through which a packet is logically sent, etc. Insome embodiments, the forwarding table entries that cause the packetprocessor 560 to send an observation to controller 500 specify to writecertain data to the registers 570 and then send a copy of the packetalong with the register contents to the controller 500.

A more detailed description of a MFE of some embodiments can be found inU.S. Publication 2013/0058250, which is incorporated by reference above.One of ordinary skill in the art will recognize that both the networkcontroller and the MFE of some embodiments includes various additionalmodules not shown in FIG. 5.

FIG. 6 conceptually illustrates a process 600 performed by the networkcontroller of some embodiments in order to execute a packet tracingoperation. The process 600 is performed, in some embodiments, by anetwork controller such as that shown above in FIG. 5. As shown, theprocess 600 begins by receiving (at 605) a command to insert a testpacket with specified source and destination addresses on a logicalforwarding element into the physical network managed by the controller.In some embodiments, the controller requires that the source anddestination addresses be connected to the same logical L2 switch (e.g.,that they not be located on different L2 switches connected by arouter). However, the source or destination address are not necessarilyboth VMs. For instance, the packet could be a broadcast packet, or couldhave as a source or destination a logical port that connects to anexternal network (e.g., via a gateway). As described above, the requestmay additionally include a timeout duration, a payload for the packet,or other parameters.

Next, the process 600 generates (at 610) a packet with the specifiedsource and destination addresses. In some embodiments, these addressesare MAC addresses that represent VMs (more specifically, virtualinterfaces (VIFs) of VMs), gateways that connect the logical switch toexternal networks, connections to a different logical switch in adifferent controller domain, etc. The controller logic has the abilityto generate a packet that mimics packets arriving at a MFE from thesource address with the destination address as the intended recipient.In addition, the controller stores an indicator in the packet (e.g., inthe packet header) that denotes the packet as a traced packet. In someembodiments, the controller uses a specific bit in the logical VLANportion of the header. Furthermore, some embodiments append to the endof the payload additional information regarding the packet tracingoperation, such as a controller identifier that uniquely identifies thecontroller issuing the packet and a tracing operation identifier thatuniquely identifies the particular trace operation issued by thecontroller.

The process then sends (at 615) the generated packet to the edge MFEassociated with the source of the packet. In some embodiments, thisbinding information (of logical port to physical port of an MFE) isstored by the network controller along with the binding of MAC addressto logical port. In some embodiments, a particular logical port (andthus source address) could reside on multiple MFEs. While a VM will beconnected to one MFE, a gateway to an external network or a connectionto a different logical switch domain might connect to several differentMFEs (e.g., all of the pool nodes in the physical domain). In this case,some embodiments allow the user to specify to which particular physicalMFE the packet will be sent. When the user does not specify one of theMFEs, the controller chooses one (e.g., randomly, from an ordered list,etc.) in some embodiments. When sending the packet to the edge MFE, insome embodiments the controller also accesses the registers (e.g., via acommand over the communications channel with the MFE) associated withthe packet on the MFE. The controller, in some embodiments, setsregister values at the MFE that mimic those for a packet received fromthe specified source address. The behavior of the edge MFE (as well asthe other MFEs that process the traced packet) will be described ingreater detail below in sub-section I.B.

The process 600 next receives (at 620) a set of observation messagesfrom MFEs in the physical network regarding the generated packet. Aswill be described in further detail below, the observation messages aresent to the network controller when an MFE performs a set of logical orphysical forwarding operation. Some of the logical forwarding operationsafter which the MFE sends observation messages include a logical L2ingress ACL operation, a logical L2 forwarding operation, a logical L2egress ACL operation, a logical L3 ACL operation, a logical L3 routingoperation, and a logical L3 egress ACL operation. For the physicalforwarding operations, some embodiments send an observation message wheneither of two actions is taken by an MFE: (1) the MFE sends the packetto another MFE via a tunnel, or (2) the MFE delivers the packet to alogical port.

Different observations provide different information for the differentoperations performed by the MFE. For instance, when the packet isdropped as a result of performing a logical L2 ingress ACL, theobservation includes some or all of the entries in the ACL, the securitypolicy (e.g., port security) that was implemented by the ACL, theidentity of the logical forwarding element to which the ACL belongs,etc. For forwarding via a tunnel, the observation includes an indicatorof the MFE performing the action and a tunnel identifier in someembodiments. From this information the controller can determine a remoteIP for the tunnel using the network state data that it stores. This maybe the remote IP of a port at the destination MFE of the tunnel. In someembodiments, the tunnel may also end at an entity in an external network(e.g., if the logical port to which the packet is sent corresponds to aremote IP in an external network). For delivery to a logical port, theobservation of some embodiments includes an indicator of the MFEperforming the action and a physical port through which the packet wouldbe delivered if it was not a traced packet. From this information, thecontroller can determine a logical port identifier (e.g., of a gateway,a VM, etc.) to which the packet is sent based on the mappings stored inits network state data.

Finally, the process 600 generates (at 625) a report based on thereceived observation messages, then ends. The report, in someembodiments, indicates whether the packet successfully reached itsdestination(s), to which destinations the packet was delivered, packetinformation specified in the request (source/destination addresses,frame size, timeout duration, etc.), the number of observationsreceived, how many times the packet was forwarded, and a time stamp thatindicates when the traced packet was sent by the controller.

In addition, for each observation received, some embodiments includeadditional information. This information may include whether theobservation was sent for forwarding (into a tunnel) or delivery to adestination, information about the source and destination MFE for aparticular tunnel, time duration from when the packet was sent by thecontroller to when the observation was received (which is useful formeasuring the latency of the network), remote IP address (forobservations indicating that the packet was forwarded into a tunnel),and logical port to which the packet would have been delivered (forobservations indicating delivery).

B. MFE Operation

The above sub-section I.A. describes the behavior of the single networkcontroller to perform a packet tracing operation, as well as thestructure of the MFEs that send observations. As mentioned, the singlenetwork controller inserts a traceable packet into the network at aparticular MFE and then receives observation messages from the MFEsthrough which the packet passes. The following section describes infurther detail the operation of the MFEs to process the packets andgenerate the observation messages.

As mentioned above, an MFE of some embodiments performs one or morelogical or physical forwarding operations and sends an observationmessages after performing a logical or physical forwarding operation.The next two figures, FIGS. 7 and 8, illustrate a MFE that sendsobservation messages after performing physical forwarding operations.Section II will describe a MFE that sends observation messages afterperforming logical forwarding operations as well as physical forwardingoperations.

FIG. 7 conceptually illustrates the processing performed by a MFE 700that receives a packet from a network controller, processes the packet,sends the packet out over a tunnel, and sends an observation to thenetwork controller. This figure illustrates the processing over fourstages 705-720.

The MFE 700 of some embodiments may be a software forwarding elementthat runs on or within a hypervisor on a VM host in a virtualizedenvironment. That is, in some embodiments the MFE 700 is one piece ofsoftware that runs in the same physical machine as VMs that connect tothe physical network through the MFE 700. In some embodiments, the MFE700 may be either a software or hardware forwarding element thatoperates on a dedicated box, separate from the machines hosting VMs. Forinstance, the pool nodes of some embodiments are dedicated boxes withsoftware or hardware forwarding elements for performing the pool nodefunctionality.

As shown at the first stage 705, the MFE 700 includes one or moreforwarding tables 725 and one or more registers 730. The forwardingtables 725, in some embodiments, include forwarding entries pushed downto the MFE 700 from a network controller as physical control plane dataand then converted into physical forwarding plane data by the MFE. Theforwarding tables 725, in some embodiments, include entries for severalunaffiliated logical networks that are virtualized over the physicalnetwork. For instance, if the machine on which the MFE 700 resides hostsVMs belonging to seven different logical networks, then the forwardingtables 725 include entries for these seven different networks, whilemaintaining isolation between the networks.

The registers 730 of the MFE 700 serve as temporary, quick-accessstorage for information while processing a packet. As described indetail in U.S. Publication 2013/0058250, which is incorporated byreference above, some embodiments resubmit a packet to the forwardingtables of the MFE several times in order to fully process the packet.During this processing, the MFE 700 uses the registers 730 totemporarily store information (e.g., logical context information, packettracing information, etc.) about the packet. In addition, someembodiments send the contents of the registers to the controller as partof an observation message.

At the first stage 705, the MFE 700 receives a packet 735 directly froma network controller that manages the MFE. As described above, in someembodiments the network controller uses the same channel to send thispacket as to send flow entries. In other embodiments, the packet is sentthrough a different channel. As shown, the packet 735 includes a payloadand a set of headers. The payload may be specified by the user, or adefault payload may be used. The headers include a source MAC addressand a destination MAC address as specified by the user who requests thepacket tracing operation, among other data, in some embodiments.Furthermore, as illustrated, the headers include a trace indicator bitthat is set to “1” to indicate that the packet is a traced packet. Insome embodiments, this bit is set within a logical VLAN field of theheader. In addition, appended to the payload (either within the payloadportion of the packet or after the payload portion), the packet 735includes trace operation data. This trace operation data includes, insome embodiments, a controller ID that identifies the issuing controlleras well as a trace operation session ID that identifies the particulartrace operation that issued the packet. In some cases, multiple tracepackets for multiple trace operations may be issued, and the trace dataallows the controller to identify to which trace operation a particularobservation applies.

Furthermore, along with the packet, the controller of some embodimentssends a command to set register information so as to mimic the receivingof the packet through a particular physical port associated with thesource address. In the case of a packet for which the source is a VM,the register data simply indicates a physical port to which the VMconnects. When the MFE actually receives a traffic (non-trace operation)packet from the VM through this physical port (e.g., virtual interface),the MFE sets this physical port in the register. Because the tracedpacket is not actually received through this physical port, thecontroller command sets the physical port in the register so that theMFE treats the packet using the same flow entries (i.e., so the sameflow entries match the packet).

The “source” of the traced packet (as set by the user requesting thetrace) might not be a VM, however. For example, the source could be agateway that connects to an external network (e.g., an unmanagednetwork). In this case, the MFE 1300 to which the packet is sent wouldbe an extender in some embodiments that contains a port facing theexternal network. However, whereas a physical port on an edge MFE onlyconnects to a single VM in general, this port on the extender may facenumerous external networks that connect to different logical networks,which correspond to different logical ports (potentially of differentlogical switches). Accordingly, some embodiments use different physicalVLANs to bridge into different logical ports, and this VLAN data as wellas the physical port will be set in the register for trace operationpackets.

In other trace operations, the source might be a remote domain thatconnects to the local domain via a pool node via a particular tunnelwith a specific interdomain context. For these operations, thecontroller of some embodiments sets in the registers for the packet boththe physical port of the pool node over which packets from theparticular tunnel are received and the interdomain context.

In some embodiments, the controller also sets one or more bits in theregister to indicate that the packet is a trace packet received from thecontroller. VMs or other entities on the network sending packets to theMFE do not have access to the registers, so this prevents such entitiesfrom sending out their own trace packets. Without such a precaution, aVM could send numerous (e.g., thousands, millions, etc.) trace packetsto the MFE as part of a denial of service attack on a controller, asthese packets would spawn numerous observations sent to the controller.

The MFE 700 receives the packet 735 and begins processing the packetthrough its forwarding tables. As mentioned, the forwarding tablesinclude numerous flow entries (e.g., on the order of hundreds orthousands of entries), which take the form of a match and correspondingaction or set of actions. That is, if the packet matches a flow entry,the entry specifies one or more actions for the MFE to take. Inaddition, in some embodiments the flow entries have differentpriorities, such that if two or more entries are matched, the MFEperforms the action or actions for the entry with the highest priority.As shown by the thicker line in the forwarding tables 725 in the secondstage 710, at this stage a flow entry is matched that specifies tomodify a particular bit in a particular register in order to indicatethat the current packet is a traced packet. Thus, as shown, one of theregisters 730 is modified to specify this trace indicator bit. In someembodiments, this entry has a high priority so that the first operationthe MFE performs upon receiving a traced packet is to set this registerbit. In some embodiments, this bit is set by the controller specifyingregister values and therefore this flow entry is not required.

In the third stage 715, a different flow entry is matched, as shown bythe thicker line in the forwarding tables 725. This flow entry specifiesto send the packet to a different MFE via a particular tunnel. Not shownin this diagram are several processing entries that would be matched inbetween the second and third stages 710 and 715. In some embodiments, toget to the point of sending the packet out via a tunnel, the MFE wouldhave already identified the source address stored in the packet and/orphysical inport entered in the register as correlating to a particularlogical input port (and therefore a particular logical switch),performed logical L2 forwarding to identify a destination logical portthat correlates to the packet's destination address (and updated orencapsulated the packet with a logical context that includes at leastthe destination logical port), and mapped that destination logical portto a physical destination.

For instance, if the trace packet specifies a source MAC A and adestination MAC B (both corresponding to VMs), and the physical portspecified in the register is port P (which is the port of the MFE 700 towhich the VM having MAC A connects), then the MFE first maps thephysical port P to a logical port X on logical switching element Q.Because the logical switching element Q is then matched (e.g., with thisinformation stored in a register), the MFE then maps the destinationaddress B to a logical port Y on the logical switching element Q, andencapsulates the packet with the destination logical port Y. Thislogical port Y maps to a remote destination Z (e.g., a remote IP or aport of a different MFE). Based on this remote destination Z, the MFEdetermines to encapsulate the packet to be sent in a tunnel T.

At the fourth stage 720, the resubmitted packet matches a flow entry towrite data to the registers for the packet and to send an observation tothe controller based on (i) the trace indicator bit stored in theregister and (ii) the decision to send the packet out through a tunnel.As shown, the MFE sends an observation message to the network controllerfrom which it received the packet. In some embodiments, this observationmessage contains (i) the packet as received and (ii) the contents of theregisters. As such, in some embodiments the flow entry that specifies tosend the observation also specifies that the MFE first write certaindata into the registers.

This data, which is then sent as part of observation message 740,includes (i) an indicator that the observation is a forwardingobservation (i.e., that the traced packet is being sent into a tunnel),(ii) a MFE ID specifying that the MFE 700 is sending the observation,and (iii) a tunnel ID denoting the tunnel through which the packet issent. Some embodiments also send a remote IP of the MFE at which thetunnel terminates, while in other embodiments the controller determinesthis data based on its stored network state information. One of ordinaryskill in the art will recognize that different forwarding observationsmay include different information.

As indicated, the MFE 700 sends the packet 735 as received to thecontroller as part of the observation message in some embodiments. Inother embodiments, the MFE sends to the controller the packet with theencapsulations that is sent to the MFE (described below). Thecontroller, in some embodiments, only uses the packet to read the tracedata stored in the packet (e.g., the controller ID and the traceoperation session ID), and reads all other pertinent information fromthe register contents sent as the observation message.

Furthermore, the MFE 700 also sends out the packet 735 through thetunnel to another MFE (e.g., the destination MFE, a pool node thatconnects to a destination MFE, etc.), which is now further encapsulated.In addition to the previous headers, the packet is now encapsulated witha logical context (e.g., the destination logical port) and a tunnelencapsulation. In some embodiments, the MFE 700 also clears out theregisters after processing the packet 735 (e.g., deletes the registersfrom memory).

The previous example illustrates an MFE processing a packet from acontroller and sending a forwarding observation message. In the managednetworks of some embodiments, MFEs also send observations whendelivering traced packets to logical ports. FIG. 8 conceptuallyillustrates the processing performed by a MFE 800 that receives a packetfrom another MFE through a tunnel, processes the packet, identifies thatthe packet should be delivered to a logical port from the MFE, and sendsan observation to the network controller. This figure illustrates theprocessing over four stages 805-820.

As with the MFE 700, the MFE 800 of some embodiments may be a softwareforwarding element that runs on or within a hypervisor on a VM host in avirtualized environment. In this case, the MFE operates within the samemachine that hosts the destination VM for the processed packet. However,delivery observations can also be generated for the delivery of packetsto external network gateways, logical routers, interconnects to adifferent logical switch managed by a different controller, or otherentities that connect to a logical port of a logical switch.

As shown at the first stage 805, the MFE 800 includes one or moreforwarding tables 825 and one or more registers 830. Like the forwardingtables of the MFE 700, the forwarding tables 825 of some embodimentsinclude forwarding entries pushed down to the MFE 800 from a networkcontroller as physical control plane data and then converted intophysical forwarding plane data by the MFE. The forwarding tables 825, insome embodiments, include entries for several unaffiliated logicalnetworks that are virtualized over the physical network. For instance,if the machine on which the MFE resides hosts VMs belonging to fivedifferent logical networks, then the forwarding tables 825 includeentries for these five different networks, while maintaining isolationbetween the networks.

Like the registers of the MFE 700, the registers 830 serve as temporary,quick-access storage for information while processing a packet. Someembodiments resubmit a packet to the forwarding tables of the MFE(through a dispatch port) several times in order to fully process thepacket. During this processing, the MFE 800 uses the registers 730 totemporarily store information (e.g., logical context information, packettracing information, etc.) about the packet. In addition, someembodiments send the contents of the registers to the controller as partof an observation message.

At the first stage 805, the MFE 800 receives a packet 835 through atunnel from a different MFE. The packet 835 is similar to the packet 735after the fourth stage 720 of FIG. 7, in that the packet is encapsulatedwith a logical context and tunnel information. While the tunnel throughwhich the packet is sent terminates at two MFEs, in between these twoMFEs may be one or more unmanaged forwarding elements that make up aphysical network over which packets are sent. In addition, like thepacket of the previous figure, the packet 835 includes a payload and aset of headers. As illustrated, the headers include a trace indicatorbit that is set to “1” to indicate that the packet is a traced packet.In some embodiments, this bit is set within a logical VLAN field of theheader. While shown as a single bit, some embodiments use more than onebit in the logical VLAN field or a different location of the packet toindicate that a packet is traced. In addition, appended to the payload(either within the payload portion of the packet or after the payloadportion), the packet 835 includes trace operation data. As describedabove for FIG. 7, this trace operation data includes, in someembodiments, a controller ID that identifies the issuing controller aswell as a trace operation session ID that identifies the particulartrace operation that issued the packet. Because this packet is receivedthrough a tunnel and not from a controller, no register data is setinitially. Instead, the physical input port is identified (because thepacket actually is received through a physical port as network trafficin this case) and stored in a register.

The MFE 800 receives the packet 835 and begins processing the packetthrough its forwarding tables. As mentioned, the forwarding tablesinclude numerous flow entries (e.g., on the order of hundreds orthousands of entries), which take the form a match and correspondingaction. As shown by the thicker line in the forwarding tables 825 in thesecond stage 810, at this stage a flow entry that is matched specifiesto modify a particular bit in a particular register in order to indicatethat the current packet is a traced packet. Thus, as shown, one of theregisters 830 is modified to specify this trace indicator bit.

In the third stage 815, a different flow entry is matched, as shown bythe thicker line in the forwarding tables 825. This flow entry specifiesto deliver the packet to its destination logical port. Not shown in thisdiagram are several processing entries that would be matched in betweenthe second and third stages 810 and 815. In some embodiments, to get tothe point of delivering the packet to a logical port, the MFE would havealready removed the tunnel encapsulation, read the logical context intoa register, etc.

At the fourth stage 820, the resubmitted packet matches a flow entry towrite data to the registers for the packet, send an observation to thecontroller based on (i) the trace indicator bit stored in the registerand (ii) the decision made at the third stage 815 to deliver the packetto its destination logical port, and to drop the packet (i.e., notactually deliver the packet to the logical port). As shown, the MFEsends an observation message to the network controller that issued thepacket (in some embodiments, all of the MFEs within a single domain aremanaged by one controller). In some embodiments, this observationmessage contains (i) the packet as generated by the controller (i.e.,without the logical context and tunnel encapsulation) and (ii) thecontents of the registers for the packet. As such, in some embodimentsthe flow entry that specifies to send the observation also specifiesthat the MFE first write certain data into the registers.

This data, which is then sent as part of the observation message 840,includes (i) an indicator that the observation is a delivery observation(i.e., that the traced packed would have been successfully delivered toa logical port if it was a real data packet), (ii) a MFE ID specifyingthat the MFE 800 is sending the observation, and (iii) a physical portID denoting the port to which the packet would have been delivered. Someembodiments also send the destination logical port, while in otherembodiments the controller determines this data based on its storednetwork state information.

As indicated, the MFE 800 sends the packet 835 to the controller as partof the observation message. In some embodiments, the MFE sends thepacket without the encapsulations (i.e., just the original headers,payload, and trace operation data). In other embodiments, the MFE sendsto the controller the packet with the encapsulations as received fromthe previous MFE. The controller, in some embodiments, only uses thepacket to read the trace data stored in the packet (e.g., the controllerID and the trace operation session ID), and reads all other pertinentinformation from the register contents sent as the observation message.Furthermore, as noted, the MFE does not deliver the packet to itsdestination, because it is only a test packet. Because the packet isinserted at a first MFE and dropped by a second MFE, the test packetsare never seen by the endpoints (e.g., the VMs), and thus do not affectthe user workloads in any way.

C. Example Packet Tracing Operations

The above sub-sections I.A. and I.B. described in detail the operationof the network controllers and MFEs of some embodiments. Thissub-section I.C. describes several example packet-tracing operations andthe observations sent to the controller for these operations. In theseexamples, the packets are all delivered to their endpoint, in order toillustrate the full scope of observations sent for various examples.However, one of ordinary skill in the art will recognize that in thesituations, for various different reasons (e.g., corrupted flow entriesin the forwarding tables of one or more MFEs), the packets may not befully delivered, in which case some of the observations may not be sent.

FIG. 9 conceptually illustrates an example for a traced packet that isgenerated and injected by the single network controller and then is sentthrough a pool node between the source MFE and destination MFE. Asshown, this example illustrates a controller 905, a first edge MFE 910,a second edge MFE 915, and a pool node 920 (also an MFE). In addition,the example illustrates various packets and observations with encirclednumbers to indicate an order in which the various data (packets,observation messages) are sent within the network.

As shown by the encircled 1, initially the controller 905 receives arequest (e.g., as described above by reference to FIG. 5). In this case,the request specifies a source logical port that maps to a VM connectedto the first MFE 910 and a destination logical port that maps to a VMconnected to the second MFE 915. In response to the request, thecontroller 905 generates a packet 925 and sends this to the MFE 910(shown by the encircled 2), along with a command to process the packet.The header of the packet 925 specifies the source and destinationaddresses received in the request, as well as a trace indicator thatmarks the packet as a test packet for a trace operation.

The first edge MFE 910 processes the packet (e.g., as shown above inFIG. 7), and its forwarding table entries in some embodiments indicateto forward the packet to the pool node 920. In some embodiments, thismay be because the edge MFE 910 does not have a direct tunnel connectionto the edge MFE 915. In this case, the edge MFE 910 (i) encapsulates thepacket with a logical context tag that specifies a destination logicalport corresponding to the destination MAC address and (ii) furtherencapsulates the packet using tunnel encapsulation with an IP addressfor a port of the pool node 920. Furthermore, the forwarding tableentries also specify to send an observation message to the controllerbecause a register bit at the MFE has been modified to identify thepacket as associated with a trace operation. Thus, as shown by theencircled 3, the first MFE 910 sends both an observation 930 to thecontroller 905 and a packet 935 to the pool node 920. In someembodiments, this is a forwarding observation message that specifies theID of the MFE 910 and the tunneling information between the edge MFE 910and the pool node 920, as described above by reference to FIG. 7. Thepacket 935 is the same as the packet 935 in some embodiments, with theadditional encapsulations mentioned above.

The pool node 920 of some embodiments receives the packet, sets aregister bit based on the trace indicator stored in the packet, removesthe tunnel encapsulation, and identifies that the destination logicalport is located at the edge MFE 915. Thus, much like the edge MFE 910,the pool node encapsulates the packet in a tunnel (this time terminatingat the edge MFE 915). As shown by the encircled 4, the pool node 920sends both an observation 940 to the controller 905 and a packet 945 tothe edge MFE 915. In some embodiments, like the observation 930, this isa forwarding observation message that specifies the ID of the pool node920 and the tunneling information between the pool node 920 and the edgeMFE 915, as described above by reference to FIG. 7. The packet 945 isthe same as the packet 935 in some embodiments, with the differenttunnel encapsulation.

The MFE 915 receives this packet 945, which still includes the traceindicator, and is encapsulated with the destination logical port. Thereceiving MFE 915 reads this destination logical port and identifies(via its forwarding table entries) to deliver the packet to the VMassociated with the destination logical port. However, based onadditional entries that (i) modify a register at the MFE to indicate thetrace packet and (ii) read this register when delivering to the logicalport, the MFE 915 instead drops the packet and sends an observation 950(indicated by the encircled 5) to the controller 905. In someembodiments, this is a delivery observation message that specifies theID of the MFE 915 and the port to which the packet would have beendelivered, as described above by reference to FIG. 8.

In the above example, the traced packet specifies a singular destinationaddress. FIG. 10 conceptually illustrates an example for a tracedbroadcast packet. As shown, this example illustrates a controller 905, afirst edge MFE 910, a second edge MFE 915, a third MFE 1020 and a poolnode 1020 (also an MFE). In addition, as with the previous figure, theexample illustrates various packets and observations with encirclednumbers to indicate an order in which the various data (packets,observation messages) are sent within the network. In this case, thelogical switch being examined has only three logical ports correspondingto three VMs, located at the hosts on which the three edge MFEs1010-1020 operated.

As shown by the encircled 1, initially the controller 1005 receives arequest (e.g., as described above by reference to FIG. 5). In this case,the request specifies a source logical port that maps to a VM connectedto the first MFE 1010. In addition, rather than specifying a particulardestination logical port, the request specifies to send a broadcastpacket to the logical switch. In response to the request, the controller1005 generates a broadcast packet 1030 and sends this to the MFE 1010(shown by the encircled 2), along with a command to process the packet.The header of the packet 1030 specifies the source address and thebroadcast destination received in the request, as well as a traceindicator that marks the packet as a test packet for a trace operation.

The first edge MFE 1010 processes the packet (e.g., as shown above inFIG. 7), and its forwarding table entries in some embodiments indicateto forward broadcast packets to the pool node 1025. Furthermore, theforwarding table entries also specify to send an observation message tothe controller because a register bit at the MFE 1010 has been modifiedto identify the packet as associated with a trace operation. Thus, asshown by the encircled 3, the first MFE 1010 sends both an observation1035 to the controller 1005 and a packet 1040 to the pool node 1025. Insome embodiments, this is a forwarding observation message thatspecifies the ID of the MFE 1010 and the tunneling information betweenthe edge MFE 1010 and the pool node 1025, as described above byreference to FIG. 7. The packet 1040 is the same as the packet 1040 insome embodiments, with the additional tunneling encapsulation mentionedabove.

The pool node 1025 of some embodiments receives the packet, sets aregister bit based on the trace indicator stored in the packet, removesthe tunnel encapsulation, and identifies the packet as a broadcastpacket. In some embodiments, the pool node 1025 converts the broadcastpacket into a set of unicast packets with the different destinations ofthe MAC addresses connected to the logical switch (excluding the sourceaddress). For each of these unicast packets, the pool node 1025 (i)encapsulates the packet with a logical context tag specifying thedestination logical port for the packet and (ii) further encapsulatesthe packet using tunnel encapsulation with an IP address for a port ofthe destination MFE at which the logical port is located. Thus, as shownby the encircled 4 and 5, the pool node sends (i) a packet 1045 to theedge MFE 1015 and a corresponding observation 1050 to the controller1005, and (ii) a packet 1055 to the edge MFE 1020 and a correspondingobservation 1060 to the controller 1005. Each of these observations is aforwarding observation indicating the respective tunneling data to theMFEs.

Each of the MFEs 1015 and 1020 receive their respective packets 1045 and1055, which still include the trace indicator and are encapsulated withtheir respective destination logical ports. The receiving MFEs 1015 and1020 read the destination logical ports and identify (via theirforwarding table entries) to deliver their packets to the VMs associatedwith the destination logical ports. However, based on additional entriesat each of the MFEs that (i) modify a register at the MFE to indicatethe trace packet and (ii) read this register when delivering to thelogical port, the MFEs 1015 and 1020 instead drop the packets and sendsobservations 1065 and 1070 (indicated by the encircled 6 and 7) to thecontroller 1005. In some embodiments, each of these are deliveryobservation messages that specify the ID of the MFE 1015 and 1020respectively, and the ports to which the packets would have beendelivered, as described above by reference to FIG. 8.

II. Tracing by Multiple Network Controllers

The above section described examples of tracing operation performed by asingle network controller that generates and injects a trace packet andreceives the observations from the MFEs to generate a report. Thissection describes examples of tracing operation performed by acontroller cluster that includes several logical controllers andphysical controllers. As mentioned above, the logical controllers ofsome embodiments generate trace packets and the physical controllers ofsome embodiments receive the observations from the MFEs that theymanage.

A. Logical and Physical Controllers

FIG. 11 conceptually illustrates a logical controller 1100 and aphysical controller 1135 as well as a MFE 1175. The logical controller1100 and the physical controller 1135 represent controllers in acontroller cluster, such as the controller cluster 305 described aboveby reference to FIG. 3. As mentioned above, the network controllers in acontroller cluster form a hierarchy. For a tracing operation, a logicalcontroller is at the top of the hierarchy, receiving trace requests fromthe user and generating trace packets. A physical controller is belowthe logical controllers in the hierarchy, relays the trace packets fromthe logical controllers to the MFEs, and receives the observations fromthe MFEs that the physical controller manages.

As shown, the logical controller 1100 includes an input interface 1105,an nLog state computation module 1110, a packet generator 1115, anobservation aggregator 1120, and an inter-controller interface 1125. Inaddition, the network controller 1100 includes one or more state storagedatabases 1130, which in some embodiments store input and/or output ofthe nLog state computation module 1110.

The input interface 1105 of some embodiments is similar to the inputinterface 505 of the single network controller 500 described above byreference to FIG. 5. The input interface 1105 receives input from one ormore users to define logical networks as well as requests to tracepackets sent through a set of logical switches and logical routers onsuch logical networks. For example, a user could define a logical switchshown in FIG. 1, described above, and a logical router shown in FIG. 20,that will be described further below. In addition, with such logicalswitches or routers defined and implemented within the physical networkmanaged by a controller cluster (which includes a number of MFEs), theuser could request a packet tracing operation for a packet sent from onelogical port to another logical port (e.g., two logical ports on thesame logical switch, one logical port of a first logical switch toanother logical port of a second logical switch via a logical router,etc.).

In some embodiments, a request that the user specifies is the sameregardless of whether the controller cluster includes a singlecontroller or a hierarchy of controllers. Therefore, in theseembodiments, the request received at the input interface 1105 specifiesthe same information as the request received at the input interface 500specifies—e.g., a source logical port (as a MAC address, a port UUID,etc.), a destination address or port, and other information about thepacket.

When the input interface 1105 receives a specification of a logicalnetwork, the interface of some embodiments translates this specificationinto logical control plane data that defines the logical network, andpasses this data to the nLog state computation module 1110. In someembodiments, the input interface 1105 reads this logical control planedata into input tables of the state computation module 1110. The nLogstate computation module 1110 of some embodiments includes a tablemapping engine with a set of input tables and output tables, and mapsrecords in the input tables to records in the output tables according toa set of rules. More specifically, some embodiments translate logicalcontrol plane data into logical forwarding plane data and subsequentlytranslate the logical forwarding plane data into physical control planedata that can be passed down to the MFEs that implement the logicalnetwork.

In some embodiments, the logical controller does not directly send thephysical control plane data to the MFEs. Rather, the logical controllerpropagates the physical control plane data to a set of physicalcontrollers, each of which manages one or more of the MFEs thatimplement the logical network. In some embodiments, the logicalcontroller translates the logical forwarding plane data into universalphysical control plane data to propagate to a set of physicalcontrollers. In some of these embodiments, the physical controllerscustomize the universal physical control plane data into customizedphysical control plane data before sending them to the MFEs thatimplement the logical network. The translation from logical controlplane data to logical forwarding plane data and then to universalphysical control plane data is described in U.S. Publication2013/0103817, which is incorporated above.

As mentioned above, some embodiments include flow entries for the traceoperation, which are passed to the MFEs. For instance, these entriesspecify that if the packet being allowed to access the logical switch isa traced packet, then certain data should be copied to the registers andsent as an observation message to the controller. More specifically, aflow entry might state “If the packet is allowed to access the logicalswitch through a logical ingress port and register bit J==1→copy MFE IDand the ingress ACL entries to registers and send register data withpacket to controller”. Similarly, a flow entry might state “If thepacket is disallowed to exit the logical switch through a logical egressport and register bit J==1→copy MFE ID and the egress ACL entries toregister and send register data with packet to controller”. These flowentries, like the other physical control plane data, are converted bythe MFE into physical forwarding plane data in the forwarding tables ofthe MFE, as described below.

In some embodiments, the nLog state computation module 1110 stores itsoutput state in the state storage database(s) 1130. This database 1130stores MAC address to logical port bindings, physical control plane dataoutput by the nLog state computation module 1110, and other data in someembodiments. The operations of the nLog state computation module 1110may take place at a different time than the packet tracing operations.

The packet generator 1115 receives requests to trace packets through theinput interface 1105. The packet generator 1115, in some embodiments,receives the source and destination logical ports, payloadspecification, frame size, etc., and generates a packet with theappropriate headers and payload. In addition, the packet generator 1115appends an indicator in the packet that specifies to a MFE processingthe packet that the packet is a traced packet. This signifies to the MFE(1) that it should send observations back to the controller when itperforms specific logical or physical forwarding operations on thepacket and (2) that it should drop the packet once it has made thedecision to deliver the packet to its intended recipient at thedestination address. In addition, some embodiments append an identifierfor the issuing controller (i.e., the logical controller 1100) as wellas a trace operation session ID to differentiate between multiple packettracing operations. Some embodiments append this data at the end of thepayload. After generating the packet, the controller sends the packet toa physical controller that manages the appropriate MFE (that whichconnects to the source logical port) through the inter-controllerinterface 1125.

The observation aggregator 1120 receives analyses of observations abouta traced packet. In some embodiments, the analyses of observations comefrom the physical controllers that manage the MFEs that send theobservations to the physical controllers. In some embodiments, a MFEsends a set of observations to a physical controller that manages theMFE by sending an observation whenever it performs a logical or physicalforwarding operation on the traced packet. The analyses of observationsare received at the inter-controller interface 1125 and sent to theobservation aggregate 1120.

In some embodiments, the analyses of observations are certain datadeduced from the observations by the controllers that receive theobservations from the MFEs. The observation aggregator 1120 aggregatesthese analyses of the observations and uses them to generate a report onthe requested trace operation. Performing analyses on the observationsby the physical controllers will be described further below.

In some embodiments, it is the logical controller that performs analyseson the observations. In these embodiments, the physical controllersreceiving the observations do not perform the analyses on theobservations and just relay the observations to the logical controllerthat generated the traced packet. The logical controller then performsthe analyses on the observations and generates a report for the tracedpacket.

In some embodiments, the observation aggregator 1120 generates a reportfor the traced packet. Some embodiments provide a summary report to theuser through the input interface 1105 that indicates whether the packetsuccessfully reached its destination(s), to which destinations it wasdelivered, packet information specified in the request(source/destination addresses, frame size, timeout duration, etc.), thenumber of observations received, the number of times the packet wasforwarded, and a time stamp that indicates when the traced packet wassent by the controller.

The logical controller 1100 connects to one or more physical controllersthrough its inter-controller interface 1125. Through this interface, thelogical controller sends traced packets to the physical controllers andreceives the analyses of the observations from the physical controllers.In addition, the logical controller exchanges other data with physicalcontrollers. For instance, the logical controller sends the physicalcontrol plane data or the universal control plane data to the physicalcontrollers. In some embodiments, the communication channels between thelogical controller 1100 and the physical controllers are remoteprocedure call (RPC) channels, used for the exchange of network statedata (e.g., the physical control plane data), the packets generated bythe logical controller, and the observation analysis sent to the logicalcontroller.

As shown in the middle portion of FIG. 11, the physical controller 1135includes an inter-controller interface 1145, an nLog state computationmodule 1150, a packet analyzer 1155, an observation analyzer 1160, and aMFE interface 1165. In addition, the physical controller 1135 includesone or more state storage database 1170, which in some embodimentsstores input and/or output of the nLog state computation module 1110.

The inter-controller interface 1145 facilitates data exchange betweenthe physical controller 1135 and other logical and physical controllers.In particular, the inter-controller interface 1145 receives tracepackets from one or more logical controllers. Also, the inter-controllerinterface 1156 receives universal physical control plane data orphysical control plane data from the logical controllers. As mentionedabove, the communication channels between the physical controller 1135and other controllers are RPC channels in some embodiments.

The packet analyzer 1155 receives trace packets from theinter-controller interface 1145. When the packet analyzer 1155 receivesa trace packet, the packet analyzer identifies the MFE at which toinject the trace packet. In some embodiments, the packet analyzer 1155first identifies the source logical port of the packet and looks up themappings between the logical ports and physical ports in the networkstate data stored in the state storage 1170. The packet analyzer thenfinds an MFE that has the physical port to which the source logical portof the packet is mapped, which should be one of the MFEs that it manages(as otherwise the logical controller would not have sent the packet tothe particular physical controller). In some embodiments, thecommunication with the trace packet from the logical controllerspecifies a particular MFE at which to inject the packet. The physicalcontroller 1135 sends the packet to the appropriate MFE through the MFEinterface 1165.

The observation analyzer 1160 receives observations about a tracedpacket sent by the MFEs to the physical controller 1135. In someembodiments, a MFE sends a set of observations to the physicalcontroller that manages this MFE whenever the MFE performs a logical orphysical forwarding operation on a trace packet. In addition, when a MFEdelivers a traced packet to its destination (or would deliver thepacket, if it was not a traced packet injected to the MFE by thephysical controller), the MFE sends an observation to the physicalcontroller. These observations are received at the MFE interface 1165and sent to the observation analyzer 1160.

The observation analyzer 1160 of some embodiments performs operations todeduce certain data from the received observations. For observationscontaining the results of performing logical forwarding operations(e.g., a logical L2 ingress ACL operation), the observation analyzer1160 of some embodiments extracts relevant information out of theobservation or reformats the information so that the logical controllercan generate a report by aggregating the information. The observationanalyzer 1160 sends the deduced data to the logical controller thatgenerated the trace packet, through the inter-controller interface 1145.

When the inter-controller interface 1145 receives physical control planedata, the interface passes this data to the nLog state computationmodule 1150. In some embodiments, the input interface 1150 reads thisdata into input tables of the state computation module 1150. The nLogstate computation module 1150 of some embodiments includes a tablemapping engine with a set of input tables and output tables, and mapsrecords in the input tables to records in the output tables according toa set of rules. More specifically, some embodiments identify the MFEs towhich to send the physical control plane data to so that the MFEs canfurther translate the physical control plane data into physicalforwarding plane data to implement the logical network.

When the inter-controller interface 1145 receives universal physicalcontrol plane data, the interface reads this data into input tables ofthe state computation module 1150. The nLog state computation module1150 translates universal physical control plane data into customizedphysical control plane data that can be passed down to the MFEs thatimplement the logical network. In other embodiments, the universalphysical control plane data is passed to all of the MFEs that areconnected to the physical controller 1135 without being processed by thenLog state computation module 1150. The MFEs, rather than the physicalcontrollers, perform the translation of the universal physical controlplane data into customized physical control plane data and then intophysical forwarding data.

In some embodiments, the nLog state computation module 1150 stores itsoutput state in the state storage database(s) 1170. This database 1170stores MAC address to logical port bindings, logical port to physicalport bindings, physical control plane data, and other data in someembodiments. The operations of the nLog state computation module 1150may take place at a different time than the packet tracing operations.That is, a user may initially set up a logical network, which requiresoperations by both the nLog state computation module 1110 of the logicalcontroller 1100 and the nLog state computation module 1150 of thephysical controller 1135 in order to generate flow entries forimplementing the logical network within a set of MFEs. At a later time,this user could initiate a packet tracing operation within the networkin order to test various aspects of the logical network.

In some embodiments, the physical controller 1135 connects to one ormore MFEs through its MFE interface 1165. Through this interface, thecontroller (i) distributes physical control plane data to the MFEs, (ii)sends traced packets for insertion into the physical network at aparticular MFE and (iii) receives observations regarding traced packetsfrom the MFEs in the physical network. In some embodiments, thecommunications channel between the controller and the MFE for thepurpose of the packet tracing operations is the same channel throughwhich the physical control plane entries are pushed down to the MFE(e.g., using a particular communication protocol such as OpenFlow). Insome embodiments, the communication through this interface is a TCPconnection between the controller and a specific control port of theMFE. Thus, IP reachability is required between the controller and thecontrol port of the MFE. Some embodiments use a specific VLAN forcontrol channel connectivity.

While the physical controller 1135 may communicate with one or moreMFEs, FIG. 11 displays additional detail regarding a particular one ofthe MFEs 1175, which is illustrative of all of the MFEs in someembodiments. This MFE 1175 may be an edge MFE which resides in a machinethat hosts VMs, a pool node, etc. The MFE 1175 includes a controllerinterface 1180, a packet processor 1185, a set of forwarding tables1190, a set of registers 1195, and a set of ports 1199 to other MFEs.

The controller interface 1180 enables the MFE to communicate with aphysical controller. Through this interface, the MFE 1175 receivesuniversal physical control plane data or customized physical controlplane data that it converts (e.g., using a module not shown in thisfigure) into physical forwarding plane data that populates theforwarding tables 1190. In some embodiments, the MFE 1175 receivesuniversal physical control plane data and converts it into customizedphysical control plane data using an nLog state computation modulelocated at the MFE. This customized physical control plane data is thenconverted into physical forwarding plane data by the MFE. The physicalforwarding plane data implements the logical forwarding operations aswell as physical forwarding operations, in some embodiments. Inaddition, through the controller interface 1180, the MFE 1175 receivestraced packets for processing and sends observation messages back to thecontroller.

The packet processor 1185 receives packets (e.g., from the controllerinterface, from the ports 1175) and processes the packets using theforwarding tables 1190. The forwarding tables 1190, in some embodiments,include entries for one or more unaffiliated logical networks that arevirtualized over the physical network. Each entry includes a conditionand a corresponding set of actions to perform if the condition ismatched.

The registers 1195 are used by the packet processor 1185 as a temporarystorage when processing a packet (e.g., in memory). In some embodiments,the packet processor 1185 uses one or more registers per packet, suchthat numerous registers may be created at a single time when numerousdifferent packets are being processed. The temporary information mayinclude an indication that a packet is for a tracing operation, resultof a logical or physical forwarding operation performed on the packet,logical context information that specifies information about the logicalforwarding element through which a packet is logically sent, etc. Insome embodiments, the forwarding table entries that cause the packetprocessor 1185 to send an observation to a physical controller specifyto write certain data to the registers 1195 and then send a copy of thepacket along with the register contents to the physical controller thatmanages the MFE 1175.

FIG. 12 conceptually illustrates a process 1200 performed by a logicalcontroller of some embodiments in order to execute a packet tracingoperation. The process 1200 is performed, in some embodiments, by alogical controller such as that shown above in FIG. 11. As shown, theprocess 1200 begins by receiving (at 1205) a command to insert a testpacket with specified source and destination addresses on a set oflogical forwarding elements (e.g., logical switches and logical routers)into the physical network implementing the logical forwarding elements.The source or destination address are not necessarily both VMs. Forinstance, the packet could be a broadcast packet, or could have as asource or destination a logical port that connects to an externalnetwork (e.g., via a gateway). As described above, the request mayadditionally include a timeout duration, a payload for the packet, orother parameters.

Next, the process 1200 generates (at 1210) a packet with the specifiedsource and destination addresses. In some embodiments, these addressesare MAC addresses that represent VMs (more specifically, VIFs of VMs) onthe same logical switch or on different logical switches, gateways thatconnect the logical switch to external networks, connections on onelogical switch to a different logical switch or to a logical router,etc. The controller logic has the ability to generate a packet thatmimics packets arriving at a MFE from the source address with thedestination address as the intended recipient. In addition, the logicalcontroller stores an indicator in the packet (e.g., in the packetheader) that denotes the packet as a traced packet. In some embodiments,the logical controller uses a specific bit in the logical VLAN portionof the header. Furthermore, some embodiments append to the end of thepayload additional information regarding the packet tracing operation,such as a controller identifier that uniquely identifies the logicalcontroller issuing the packet and a tracing operation identifier thatuniquely identifies the particular trace operation issued by the logicalcontroller.

The process then sends (at 1215) the generated packet to a physicalcontroller that manages the edge MFE associated with the source of thepacket. In some embodiments, this binding information (of logical portto physical port of an MFE) is stored by the logical controller alongwith the binding of MAC address to logical port. The logical controlleralso stores a list of a mapping of MFEs to their managing physicalcontroller. The physical controller subsequently injects the packet intothe MFE associated with the source of the packet.

In some embodiments, a particular logical port (and thus source address)could reside on multiple MFEs. While a VM will be connected to one MFE,a gateway to an external network or a connection to a different logicalswitch domain might connect to several different MFEs (e.g., all of thepool nodes in the physical domain). In this case, some embodiments allowthe user to specify to which particular physical MFE the packet will besent. When the user does not specify one of the MFEs, the controllerchooses one (e.g., randomly, from an ordered list, etc.) in someembodiments.

The process 1200 next receives (at 1220) a set of analyses ofobservation messages from a set of physical controllers. These physicalcontrollers manage the MFEs through which the trace packet passes. Asdescribed above, the MFEs send observation messages to the physicalcontrollers that manage the MFEs whenever the MFEs perform logicaland/or physical forwarding operations. The physical controllers analyzesthe observation messages and sends the analyses to the logicalcontroller that generated the trace packet.

Finally, the process 1200 generates (at 1225) a report based on thereceived analyses, then ends. The report, in some embodiments, indicateswhether the packet successfully reached its destination(s), to whichdestinations the packet was delivered, packet information specified inthe request (source/destination addresses, frame size, timeout duration,etc.), the number of observations received, how many times the packetwas forwarded, and a time stamp that indicates when the traced packetwas sent by the controller.

In addition, for each observation received by the controller cluster,some embodiments include additional information. For physical forwardingobservations, this information may include whether the observation wassent for forwarding (into a tunnel) or delivery to a destination,information about the source and destination MFE for a particulartunnel, time duration from when the packet was sent by the controller towhen the observation was received, remote IP address (for observationsindicating that the packet was forwarded into a tunnel), and logicalport to which the packet would have been delivered (for observationsindicating delivery). For logical processing observations, thisinformation might include ACL decisions (e.g., why a packet wasdropped), the logical port to which a packet was forwarded on aparticular logical switch or logical router, etc. Some of the logicalprocessing observations of some embodiments are described in greaterdetail below by reference to FIG. 21.

FIG. 13 conceptually illustrates a process 1300 performed by the networkcontroller of some embodiments in order to execute a packet tracingoperation. The process 1300 is performed, in some embodiments, by aphysical controller such as that shown above in FIG. 11. As shown, theprocess 1300 begins by receiving (at 1305) from a logical controller atest packet with specified source and destination addresses on a set oflogical network element (e.g., logical switches and logical routers). Asdescribed above, the logical controller generates the packet from atrace request received from a user.

Next, the process 1300 identifies (at 1310) a MFE to which to inject thepacket. In some embodiments, the process 1300 uses the source MACaddress of the packet and mappings between the physical ports and theaddresses of VMs. In some embodiments, these addresses are MAC addressesthat represent VMs (more specifically, VIFs of VMs), gateways thatconnect the logical switch to external networks, connections to adifferent logical switch in a different controller domain, etc. In otherembodiments, the communication from the logical controller with the testpacket additionally includes a specification of the MFE at which thepacket should be inserted.

The process then sends (at 1315) the packet to the edge MFE associatedwith the source of the packet. In some embodiments, this bindinginformation (of logical port to physical port of an MFE) is stored bythe network controller along with the binding of MAC address to logicalport. When sending the packet to the edge MFE, in some embodiments thephysical controller also accesses the registers (e.g., via a commandover the communications channel with the MFE) associated with the packeton the MFE. The controller, in some embodiments, sets register values atthe MFE that mimic those for a packet received from the specified sourceaddress.

The process 1300 next receives (at 1320) a set of observation messagesfrom MFEs in the physical network regarding the injected packet andanalyzes the observation messages to deduce certain data from themessages. The observation messages are sent to the network controllerwhen an MFE performs a set of logical or physical forwarding operations.Some of the logical forwarding operations after which the MFE sendsobservation messages include a logical L2 ingress ACL operation, alogical L2 forwarding operation, a logical L2 egress ACL operation, alogical L3 ACL operation, a logical L3 routing operation, and a logicalL3 egress ACL operation. For the physical forwarding operations, someembodiments send an observation message when either of two actions istaken by an MFE: (1) the MFE sends the packet to another MFE via atunnel, or (2) the MFE delivers the packet to a logical port.

The physical controller receives different observations that providedifferent information for the different operations performed by theMFEs. For instance, when the packet is dropped as a result of performinga logical L2 ingress ACL, the observation includes some or all of theentries in the ACL, the security policy (e.g., port security) that wasimplemented by the ACL, the identity of the logical forwarding elementto which the ACL belongs, etc. For forwarding via a tunnel, theobservation includes an indicator of the MFE performing the action and atunnel identifier in some embodiments. From this information thephysical controller can determine a remote IP for the tunnel port at thedestination MFE of the tunnel using the network state data that itstores. For delivery to a logical port, the observation of someembodiments includes an indicator of the MFE performing the action and aphysical port through which the packet would be delivered if it were nota traced packet. From this information, the physical controller candetermine a logical port identifier (e.g., of a gateway, a VM, etc.) towhich the packet is sent based on the mappings stored in its networkstate data.

Finally, the process 1300 sends (at 1325) the analyses of theobservation messages to the logical controller. As mentioned above, theobservation messages include the identification of the logicalcontroller that generated the trace packet. The process thus identifiesthe logical controller to which to send the analyses to and sends themto the identified logical controller.

It will be noted that this process 1300 is performed by a singlephysical controller for a given trace operation (specifically, thecontroller that manages the MFE at which the trace packet is inserted).However, the operations 1320 and 1325 are only performed by thisparticular physical controller for observations sent by MFEs that thisphysical controller manages. For the same trace operation, otherphysical controllers will perform operations 1320 and 1325 forobservation messages received from MFEs that these controllers manageand then send the analyses to the same logical controller.

B. Example Packet Tracing Operations

The above sub-section II.A. describes in detail the operation of logicalcontrollers and physical controllers of some embodiments. Thissub-section II.B. describes several example packet-tracing operationsand the observations sent to a controller cluster that includes a set oflogical controllers and physical controllers.

FIG. 14 conceptually illustrates an example for a traced packet that isgenerated by a logical controller and injected to a MFE by a physicalcontroller. This traced packet is sent through a pool node between thesource MFE and destination MFE. As shown, this example illustrates acontroller cluster 1400, a first edge MFE 1420, a second edge MFE 1425,and a pool node 1430 (also an MFE). In addition, the example illustratesvarious packets and observations with encircled numbers to indicate anorder in which the various data (packets, observation messages) are sentwithin the network. As shown, the controller cluster 1400 in thisexample includes a logical controller 1405 and two physical controllers1410 and 1415.

As shown by the encircled 1, initially the logical controller 1405receives a request. In this case, the request specifies a source logicalport that maps to a VM connected to the first MFE 1420 and a destinationlogical port that maps to a VM connected to the second MFE 1425. Inresponse to the request, the controller 1405 generates a packet 1425.

As shown by the encircled 2, the logical controller 1405 then sends thegenerated packet to the physical controller 1410 because the physicalcontroller 315 manages the MFE 1420 associated with the source of thepacket. The header of the packet 1435 specifies the source anddestination addresses received in the request, as well as a traceindicator that marks the packet as a test packet for a trace operation.

The physical controller 1410 identifies that the MFE 1420 is the MFEinto which to inject the packet (e.g., based on a communication from thelogical controller 1405 or by examining the packet) because the VM towhich the destination logical port maps is connected to the MFE 1420. Asshown by the encircled 3, the physical controller 1410 sends the packet335 to the MFE 1420, along with a command to process the packet.

The first edge MFE 1420 processes the packet by performing a set oflogical and physical forwarding operations. The MFE's forwarding tableentries in some embodiments indicate to forward the packet to the poolnode 1430. In some embodiments, this may be because the edge MFE 1420does not have a direct tunnel connection to the edge MFE 1425. In thiscase, the edge MFE 1420 (i) encapsulates the packet with a logicalcontext tag that specifies a destination logical port corresponding tothe destination MAC address and (ii) further encapsulates the packetusing tunnel encapsulation with an IP address for a port of the poolnode 1430. Furthermore, the forwarding table entries also specify tosend an observation message to the managing physical controller of theMFE 1420 whenever performing a logical or physical forwarding operationbecause a register bit at the MFE has been modified to identify thepacket as associated with a trace operation. Thus, as shown by theencircled 4, the first MFE 1420 sends a set of observations 1440 to thephysical controller 1410. The first MFE 1420 sends the observations tothe physical controller 1410 because the physical controller 1410manages the first MFE 1420. While shown as one set of data, theseobservations may be sent separately as each operation is performed bythe MFE 1420.

The physical controller 1410 receives the set of observations 1440,analyzes them, and sends (shown by the encircled 5) the analyses to thelogical controller 1405. In some embodiments, the physical controller1410 uses the information (e.g., logical controller ID) included in theobservation messages to identify the logical controller to which to sendthe analyses.

As the first edge MFE 1420 sends the last observation of the set ofobservations 1440, the first edge MFE 1420 sends (shown by encircled 6)a packet 1436 to the pool node 1430. The packet 1436 is the same as thepacket 1435 in some embodiments, with the additional encapsulationsmentioned above. It is to be noted that an instance in time when thepacket 1436 is sent to the pool node 1430 is not necessarily after aninstance in time when the analyses of the set of observations 1440 aresent to the logical controller 1405, even though the encircled numbersfor the analyses and the packet are 5 and 6, respectively. The amount oftime for the set of observations 1440 to reach the physical controller1410 in addition to the amount of time the physical controller 1410spends to analyze the set of observations received from the MFE 1420 maybe longer or shorter than the amount of time for the packet 1436 to besent to and reach the pool node 1445.

The pool node 1430 of some embodiments receives the packet 1436, sets aregister bit based on the trace indicator stored in the packet, removesthe tunnel encapsulation, and identifies that the destination logicalport is located at the second edge MFE 1425. Thus, much like the edgeMFE 1420, the pool node encapsulates the packet in a tunnel (this timeterminating at the edge MFE 1425). As shown by the encircled 7, the poolnode 1430 sends its own set of observations 1445 to the physicalcontroller 1410 as the pool node 1430 performs a set of logical and/orphysical forwarding operations to send the packet to the edge MFE 1425.In this example, the physical controller 1410 manages both the edge MFE1420 and the pool node 1430. However, in many cases, the pool node willbe managed by a different physical controller than the edge MFE (e.g.,the physical controller 1415 that manages the destination MFE, or adifferent physical controller altogether). The physical controller 1410receives the set of observations 1445, analyzes them, and sends (shownby the encircled 8) the analyses to the logical controller 1405.

As the pool node 1430 sends the last observation of the set ofobservations 1445, the pool node 1430 sends (shown by encircled 8) apacket 1450 to the second edge MFE 1425. The packet 1450 is the same asthe packet 1436 in some embodiments, with the additional or differentencapsulations mentioned above. It is to be noted that an instance intime when the packet 1450 is sent to the edge MFE 1425 is notnecessarily after an instance in time when the analyses of the set ofobservations 1445 even though the encircled numbers for the analyses andthe packet are 8 and 9, respectively.

The MFE 1425 receives this packet 1450, which still includes the traceindicator, and is encapsulated with the destination logical port. Thereceiving MFE 1425 reads this destination logical port and identifies(by performing a set of logical and physical forwarding operationsspecified by the MFE's forwarding table entries) to deliver the packetto the VM associated with the destination logical port. However, basedon additional entries that (i) modify a register at the MFE to indicatethe trace packet and (ii) read this register when delivering to thelogical port, the MFE 1425 instead drops the packet.

Moreover, as the MFE 1425 performs its set of logical and physicalforwarding operations, the MFE 1425 sends (shown by the encircled 10) aset of observations 1455 to the physical controller 1415. The second MFE1425 sends the observations to the physical controller 1415 because thephysical controller 1415 manages the second MFE 1415. The physicalcontroller 1415 receives the set of observations 1455, analyzes them,and sends (shown by the encircled 11) the analyses to the logicalcontroller 1405.

The logical controller 1405 receives the analyses of the observationsfrom the physical controller 1410 and waits for more analyses to comefrom other physical controllers (e.g., the physical controller 1415)until a timeout is reached. Once the timeout is reached, the logicalcontroller 1405 generates a report for the requested tracing operationand sends the report to the user, as shown by the encircled 12.

FIG. 14 illustrates that the logical controller 1405 is at the top ofthe hierarchy formed by the controllers in the controller cluster. Insome embodiments, another type of controller resides above the logicalcontroller 1405 in the hierarchy. This controller at the top is referredto as an API controller or input translation controller. The APIcontroller, in some embodiments, supports the API of the controllercluster and takes the inputs from the user in the form of API calls. TheAPI controller then generates the request and sends the request to alogical controller which is responsible for managing the logicalforwarding elements of the user. More details about an API controller(input translation controller) are described in the above-incorporatedU.S. Publication 2013/0103817.

In the above example, the traced packet specifies a singular destinationaddress. FIG. 15 conceptually illustrates a second example for a tracedbroadcast packet that is generated by a logical controller and injectedto a MFE by a physical controller. As shown, this example illustrates alogical controller 1505, three physical controllers 1510-1520, a firstedge MFE 1525, a second edge MFE 1535, a third MFE 1540 and the poolnode 1530 (also an MFE). In addition, as with the previous figure, theexample illustrates various packets and observations with encirclednumbers to indicate an order in which the various data (packets,observation messages) are sent within the network. In this case, thelogical switch being examined has only three logical ports correspondingto three VMs, located at the hosts on which the three edge MFEs 1525,1535, and 1540 operate.

As shown by the encircled 1, initially the logical controller 1505receives a request to trace a specified packet from a source logicalport. In this case, the request specifies a source logical port thatmaps to a VM connected to the first MFE 1525. In addition, rather thanspecifying a particular destination logical port, the request specifiesto send a broadcast packet to the logical switch (e.g., by specifying abroadcast address on the logical switch as the packet destination). Inresponse to the request, the controller 1505 generates a broadcastpacket 1545.

As shown by the encircled 2, the logical controller 1505 then sends thegenerated packet to the physical controller 1510 because the physicalcontroller 1510 manages the MFE 1525 associated with the source of thepacket. The header of the packet 1545 specifies the source address andthe broadcast destination received in the request, as well as a traceindicator that marks the packet as a test packet for a trace operation.

The first edge MFE 1525 processes the packet by performing a set oflogical and physical forwarding operations. The MFE 1525's forwardingtable entries in some embodiments indicate to forward broadcast packetsto the pool node 1525. Furthermore, the forwarding table entries alsospecify to send an observation message to the physical controller 1510whenever performing a logical or physical forwarding operation because aregister bit at the MFE 1525 has been modified to identify the packet asassociated with a trace operation. Thus, as shown by the encircled 3,the first MFE 1510 sends a set of observations 1550 to the physicalcontroller 1510. The first MFE 1525 sends the observations to thephysical controller 1510 because the physical controller 1510 managesthe first MFE 1525. As in the previous figure, the MFE 1525 does notnecessarily send all of the observations together, but instead someembodiments send each observation as the corresponding operation (e.g.,a logical ACL decision, a logical forwarding decision to a specificlogical port, a physical forwarding operation into a tunnel, etc.) isperformed by the MFE 1525.

The physical controller 1510 receives the set of observations 1550,analyzes them, and sends (shown by the encircled 5) the analyses to thelogical controller 1505. In some embodiments, the physical controller1510 uses the information (e.g., logical controller ID) included in theobservation messages to identify the logical controller to which to sendthe analyses.

As the first edge MFE 1525 sends the last observation of the set ofobservations 1550, the first edge MFE 1525 sends (shown by encircled 6)a packet 1555 to the pool node 1530. The packet 1555 is the same as thepacket 1545 in some embodiments, with the additional encapsulationsmentioned above. It is to be noted that an instance in time when thepacket 1555 is sent to the pool node 1530 is not necessarily after aninstance in time when the analyses of the set of observations 1550 issent to the logical controller 1505 even though the encircled numbersfor the analyses and the packet are 5 and 6, respectively. This isbecause the amount of time the set of observations 1550 to reach thephysical controller 1510 in addition to the amount of time the physicalcontroller 1510 spends to analyze the set of observations received fromthe MFE 1525 may be longer or shorter than the time for the packet 1555to be sent to the pool node 1530.

The pool node 1530 of some embodiments receives the packet, sets aregister bit based on the trace indicator stored in the packet, removesthe tunnel encapsulation, and identifies the packet as a broadcastpacket. In some embodiments, the pool node 1525 converts the broadcastpacket into a set of unicast packets with the different destinations ofthe MAC addresses connected to the logical switch (excluding the sourceaddress). For each of these unicast packets, the pool node 1525 (i)encapsulates the packet with a logical context tag specifying thedestination logical port for the packet and (ii) further encapsulatesthe packet using tunnel encapsulation with an IP address for a port ofthe destination MFE at which the logical port is located.

As shown by the encircled 7, the pool node 1530 sends its own set ofobservations 1560 to the physical controller 1510 as the pool node 1530performs a set of logical and physical forwarding operations to convertthe broadcast packet to the set of unicast packets and send the unicastpackets to the respective destination MFEs. The physical controller 1510receives the set of observations 1560, analyzes them, and sends (shownby the encircled 8) the analyses to the logical controller 1505.

As shown by the encircled 9, the pool node sends (i) a packet 1565 tothe edge MFE 1535 and (ii) a packet 1570 to the edge MFE 1540. Each ofthe corresponding observations for sending these two packets is aforwarding observation indicating the respective tunneling data to theMFEs (sent as part of the set of observations 1560).

The MFEs 1535 and 1540 receive their respective packets 1545 and 1555,which still include the trace indicator and are encapsulated with theirrespective destination logical ports. The receiving MFEs 1535 and 1540read the destination logical ports and identify (by performing a set oflogical and physical forwarding operations specified by their forwardingtable entries) to deliver their packets to the VMs associated with thedestination logical ports. However, based on additional entries at eachof the MFEs that (i) modify a register at the MFE to indicate the tracepacket and (ii) read this register when delivering to the logical port,the MFEs 1515 and 1505 instead drop the packets.

Moreover, as the MFEs 1535 and 1540 perform their respective set oflogical and physical forwarding operations, the MFEs 1535 and 1540 send(shown by the encircled 10) their respective set of observations 1555and 1580 to the physical controllers 1520 and 1515, respectively. Thesecond MFE 1535 sends the observations to the physical controller 1520because the physical controller 1520 manages the second MFE 1535. Thethird MFE 1540 sends the observations to the physical controller 1515because the physical controller 1515 manages the second MFE 1540.

The physical controller 1515 receives the set of observations 1580,analyzes them, and sends (shown by the encircled 11) the analyses to thelogical controller 1505. The physical controller 1520 receives the setof observations 1575, analyzes them, and sends (shown by the encircled11) the analyses to the logical controller 1505.

The logical controller 1505 receives the analyses of the observationsfrom the physical controller 1510 and waits for more analyses to comefrom other physical controllers (e.g., the physical controllers 1515 and1520) until a timeout is reached. Once the timeout is reached, thelogical controller 1505 generates a report for the requested tracingoperation and sends the report to the user, as shown by the encircled12.

C. Multiple Logical Controllers

In the examples described above, a logical controller that generated atrace packet received the observations or analyses of the observationsfrom a set of physical controllers that manage the MFEs through whichthe packet passes. In some embodiments, another logical controller thatdid not generate a trace packet aggregates some of the observations oranalyses of the observations resulted from processing the trace packet.This logical controller then passes the aggregated observations or theanalyses to the logical controller that generated the trace packet. Insome embodiments, this may occur in larger-scale deployments in whicheach logical controller does not have a direct connection to eachphysical controller.

FIG. 16 conceptually illustrates an example for a trace packet that isgenerated by a first logical controller. In this example, a secondlogical controller aggregates some of the observations resulted fromprocessing the trace packet and passes the observations to the firstlogical controller. The first logical controller generates a report withthe observations from the second logical controller as well as thoseobservations that the logical controller aggregates.

As shown, this figure illustrates a first logical controller 1605, asecond logical controller 1610, three physical controllers 1615-1625,and four MFEs 1630-1645. In addition, the example illustrates the pathof the trace packet with solid-head arrows and illustrates the paths ofthe observations with hollow-head arrows as indicated in the dotted boxillustrated in the bottom of the figure. The example illustratesencircled numbers to indicate an order in which the various data(packets, observation messages, analysis, etc.) are sent within thenetwork. However, when a path diverts into branches (e.g., from 3 to 4and 6) the numbers may not necessarily represent the temporal order inwhich the communications occur, owing to the potential for differenttime lags in different branches.

In some embodiments, the logical controllers share all logical andphysical data for implementing a set of logical forwarding elements inthe physical network. In some such embodiments, the logical controllerscommunicate with different sets of physical controllers, for example, inorder to scale. In this example, the logical controller 1605 propagatesphysical control plane data to the physical controllers 1615 and 1620,and the logical controller 1610 propagates physical control plane datato the physical controller 1625. Moreover, the physical controller 1615manages the MFE 1630, the physical controller 1620 manages the MFE 1635,and the physical controller 1625 manages the MFEs 1640 and 1645. One ofordinary skill in the art will recognize that each physical controller1615-1625 would generally manage numerous MFEs, which are not shown asthey are not involved in the illustrated packet tracing operations.

As shown by the encircled 1, the first logical controller 1605 generatesa trace packet from a request received from the user. The first logicalcontroller 1605 sends the trace packet to the physical controller 1615as shown by the encircled 2 because the physical controller 1615 managesthe MFE 1630 to which the source address of the trace packet connects,and the logical controller 1605 has a direct connection to the physicalcontroller 1615. If, on the other hand, the source logical port for thetrace packet mapped to a physical port of the MFE 1645, some embodimentswould send the trace packet to the logical controller 1610 forsubsequent transmission to the physical controller 1625, which managesthe MFE 1645.

The physical controller 1615 then injects the trace packet 1630 into theMFE 1630, in the manner described above. The MFE 1630 performs a set oflogical and physical forwarding operations on the trace packet whilesending (as shown by the encircled 4) an observation to the physicalcontroller 1615 whenever the MFE performs a logical or physicalforwarding operation on the packet.

The physical controller 1615 then receives the observations, analyzeseach of them, and sends the analysis of the observation to the logicalcontroller 1605. The logical controller 1605 receives the analyses andstarts preparing a report of the trace operation by aggregating theanalyses of the observation. The logical controller 1605 waits for moreanalyses to come from other physical controllers until a timeout isreached.

As shown by the encircled 6, the MFE 1630 sends the packet to MFE 1635through a tunnel. In some embodiments, the MFE 1635 is a pool node,though this MFE may also be an edge MFE in other embodiments. The MFE1635 then processes the packet by also performing a set of logical andphysical operations on the received packet. The MFE 1635 sends (as shownby the encircled 7) its own set of observations to the physicalcontroller 1620, which, in turn, analyzes this set of observations andsends (as shown by the encircled 8) the analyses to the logicalcontroller 1605.

As shown by the encircled 9, the MFE 1635 sends the packet to MFE 1640through a tunnel. The MFE 1640 then processes the packet by performing aset of logical and physical operations on the received packet. As withthe MFE 1635, this MFE may be a pool node in some embodiments and anedge MFE in other embodiments. The MFE 1640 sends (as shown by theencircled 10) its own set of observations to the physical controller1625, which analyzes this set of observations and sends (as shown by theencircled 11) the analyses to the second logical controller 1610.

The second logical controller 1610 then determines that the analysis ofthe observation is not resulted from processing a trace packet that thesecond logical controller 1610 generated. In some embodiments, a logicalcontroller determines whether an observation or the analysis of theobservation is resulted from processing a trace packet that itgenerated, based on the information (e.g., the logical controller ID)that indicates the origin of the traced packet, which is part of theobservation message received by the physical controller and thus part ofthe analysis in the multiple logical controller situation. In thisexample, the analyses of the set of observations that the logicalcontroller 1610 received from the physical controller 1625 includes thelogical controller ID of the first logical controller 1605. Therefore,the second logical controller 1610 sends (as shown by the encircled 12)the analysis or the observation to the logical controller 1605.

As shown by the encircled 13, the MFE 1640 sends the packet to MFE 1645through a tunnel. The MFE 1645 then processes the packet by performing aset of logical and physical operations on the received packet, includinga decision to deliver the packet to its destination logical port (aswell as dropping the packet before actually delivering it). The MFE 1645sends (as shown by the encircled 14) its own set of observations to thephysical controller 1625, which analyzes this set of observations andsends (as shown by the encircled 15) the analyses to the second logicalcontroller 1610.

The second logical controller 1610 again determines that the analyses ofthe observations are not the result of processing a trace packetgenerated by the second logical controller 1610. In this example, theanalyses of this set of observations that the logical controller 1610received from the physical controller 1625 includes the logicalcontroller ID of the first logical controller 1605. Therefore, thesecond logical controller 1610 sends (as shown by the encircled 16) theanalysis or the observation to the logical controller 1605. The logicalcontroller 1605 generates a report for the requested tracing operationby aggregating the analyses of the observations from the physicalcontrollers 1615 and 1620 and the logical controller 1610 and sends thereport to the user, as shown by the encircled 17.

In the example illustrated in FIG. 16, a first logical controllerreceived observations from a second logical controller because the firstlogical controller generated the trace packet the processing of whichresulted in the observation messages and analysis. FIG. 17 conceptuallyillustrates an example for two trace packets originating from twodifferent logical controllers. In this example, a first logicalcontroller aggregates some of the observations resulted from processinga first trace packet generated by a second logical controller and passesthe observations to the second logical controller. The second logicalcontroller also aggregates some of the observations resulted fromprocessing a second trace packet generated by the first logicalcontroller and passes the observations to the first logical controller.

As shown, this figure illustrates the same network control system as theprevious figure: the first logical controller 1605, the second logicalcontroller 1610, the three physical controllers 1615-1625, and the fourMFEs 1630-1645. This figure also illustrates paths of two trace packets1 and 2 generated by the first and second logical controllers 1605 and1610, respectively. The paths for the trace packet 1 and its associatedobservations are illustrated as solid-head arrows and hollow-head arrowsas indicated in the left dotted box shown in the bottom of the figure.The paths for the trace packet 2 and its associated observations areillustrated as double solid-head arrows and double hollow-head arrows,respectively, as indicated in the right dotted box shown in the bottomof the figure. The example illustrates encircled numbers representingthe first trace packet and its resulting observation messages andanalysis, and squared numbers representing the second trace packet andits resulting observation messages and analysis. The encircled andsquared numbers indicate an order in which the various data for aparticular trace operation (packets, observation messages, analysis,etc.) are sent within the network. However, when a path diverts intobranches (e.g., from the encircled 3 to the encircled 4 and 6), thenumbers may not necessarily represent the temporal order in which thecommunications occur, owing to the potential for different time lags indifferent branches. In addition, the encircled numbers representingcommunications for the first trace operation and the squared numbersrepresenting communications for the second trace operation do notnecessarily take place at the same time.

The description for the trace operation for the trace packet 1 in thisexample is identical with the description for the trace operation forthe trace packet in the example of FIG. 16 above. Also, the descriptionfor the trace operation for the trace packet 2 is similar to thedescription for the example of FIG. 16, except that the packet traversesthe set of MFEs in the opposite direction (e.g., from a source portlocated at the MFE 1645 to a destination port located at the MFE 1630)and therefore the observations/analyses are sent in this opposite orderby the MFEs and physical controllers. Furthermore, whereas in the firsttrace operation the first logical controller 1605 aggregates theanalyses (including some sent by the logical controller 1610) andgenerates a report, in the second trace operation the second logicalcontroller 1610 receives some of its analyses from the first logicalcontroller 1605 and performs the aggregation into a report to output tothe user that requested the trace operation.

III. Observations for Logical Forwarding Operations

As mentioned above, an MFE of some embodiments generates an observationafter performing a logical or physical forwarding operation. Because anMFE performs one or more logical or physical forwarding operations on apacket, the MFE sends one or more observations to a controller cluster.This section describes logical and physical forwarding operations forwhich an MFE generate observations.

A. Managed Forwarding Element Operation

The next two figures, FIGS. 18 and 19, illustrate a MFE that sendsobservation messages as it performs a set of logical and physicalforwarding operations. FIG. 18 conceptually illustrates packetprocessing performed by a MFE 1800 that receives a trace packet from anetwork controller, processes the packet, sends the packet out over atunnel, and sends several observations to the network controller. Thisfigure illustrates the processing over eight stages 1805-1820.

As with the MFE 700 described above by reference to FIG. 7, the MFE 1800of some embodiments may be a software forwarding element that runs on orwithin a hypervisor on a VM host in a virtualized environment. Moreover,the MFE 1800 is similar to MFE 700 in that the MFE 1800 includes one ormore forwarding tables 1845 and one or more registers 1850. In someembodiments, these registers are created (e.g., in the memory of theMFE) on a per packet basis, and the registers illustrated are those forthe currently processed packet 1855.

At the first stage 1805, the MFE 1800 receives a packet 1855 directlyfrom a network controller that manages the MFE. This network controllercould be a physical controller that manages the MFE or a single networkcontroller that generates packets and reports on the tracing of thepackets. As described above, in some embodiments the network controlleruses the same channel to send this packet as to send flow entries. Thepacket 1855 is similar to the packet 735 described above by reference toFIG. 7. In addition, as in the example of FIG. 7, the controller sets aregister at the MFE 1800 to include the physical inport through whichthe packet would have been received had the packet been sent from itssource address.

The MFE 1800 receives the packet 1855 and begins processing the packetthrough its forwarding tables. As mentioned, the forwarding tablesinclude numerous flow entries, which take the form of a match andcorresponding action or set of actions. As shown by the thicker line inthe forwarding tables 1845 in the second stage 1810, at this stage aflow entry that is matched specifies to modify a particular bit in aparticular register in order to indicate that the current packet is atraced packet. Thus, as shown, one of the registers 1850 is modified tospecify this trace indicator bit. In some embodiments, this entry has ahigh priority so that the first operation the MFE performs uponreceiving a traced packet is to set this register bit. In someembodiments, this bit is set by the controller specifying registervalues and therefore this flow entry is not required.

In the third stage 1815, a different flow entry is matched, as shown bythe thicker line in the forwarding tables 1845. This flow entryspecifies to determine the logical ingress context of the packet. Insome embodiments, a logical context represents the state of the packetwith respect to the logical forwarding element. Because this packet isreceived from the network controller, but made to look as though it wasreceived directly from a source address, the packet does not yet have alogical context tag. However, based on the physical inport stored in theregisters 1850 for the packet, the forwarding table maps the packet to aparticular logical ingress port of a particular logical forwardingelement (e.g., a logical switch). In some embodiments, this informationis stored in a register, as shown. However, no observation is sent atthis point, as mapping ingress context does not cause an observationmessage to be sent in some embodiments.

For simplicity of illustration, the fourth stage 1820 shows that theresubmitted packet matches one flow entry. However, the packet isresubmitted a number of times to perform a set of logical forwardingoperations specified by a set of flow entries that is matchedsequentially. Some of these logical forwarding operations include alogical ingress ACL operation for determining access control when thelogical forwarding element receives the packet, a logical L2 operationfor determining where to forward the packet through a layer 2 network,and a logical egress ACL operation for determining access control beforethe logical forwarding element routes the packet out of the logicalforwarding element. Alternatively, or in conjunction with the logical L2operations, the logical forwarding operations include a logical L3operation for determining where to route the packet through a logicallayer three network. For example, an ingress ACL operation might bebased on the logical input port stored in the registers, a logical L2forwarding operation might be based on a MAC address stored in thepacket header, etc. These operations are described in further detailbelow.

The MFE 1800 stores the result of performing each of these operations inthe registers in some embodiments (not shown). That is, some embodimentsupdate the registers for the packet each time an operation specified bya flow entry is performed. Moreover, after a logical forwardingoperation is performed, the resubmitted packet matches a flow entry thatspecifies to send an observation to the controller or controllercluster. The MFE 1800 sends an observation after performing a logicalforwarding operation. Sending of an observation after performing alogical forwarding operation is shown by the fifth stage 1825. As shown,the MFE sends an observation message to the network controller fromwhich it received the packet. In some embodiments, this observationmessage contains (i) the packet and (ii) the contents of the registers.The contents in the registers, which is then sent as part of observationmessage 1860, includes an (i) an indicator that the observation is aresult of performing a logical forwarding operation (e.g., allowing thepacket to enter the logical switch, dropping the packet as the packet issent to the logical egress port, etc.), (ii) a MFE ID specifying thatthe MFE 1800 is sending the observation, etc. The different observationssent for different logical processing operations of some embodiments aredescribed below by reference to FIG. 21.

Therefore, in order to send a set of observations to the networkcontroller, the MFE actually repeats the stages 1820 and 1825 as manytimes as the number of logical operations it performs in someembodiments (with stage 1825 repeated only for the operations thatresult in an observation message). After performing all the logicaloperations, the MFE 1805 resubmits the packet unless the packet isdropped as a result of performing a logical forwarding operation (e.g.,an ACL operation that specifies to drop the packet). In addition, whenlogical forwarding decisions are made by the MFE (e.g., forwarding thepacket to a particular destination logical port of a logical forwardingelement), some embodiments encapsulate the packet with this destinationlogical port.

In the sixth stage 1830, the packet matches a flow entry that specifiesto determine one or more ports of one or more MFEs that correspond tothe logical egress port of the logical forwarding element through whichthe packet is to be sent out. For instance, if the packet is a broadcastpacket or a multicast packet, in some embodiments the MFE 1800determines the ports of the MFEs that correspond to the logical egressports of the logical forwarding element through which the packet is tobe broadcasted or multicasted out (i.e., the logical ports to which theintended recipients of the packet is coupled). However, in otherembodiments, broadcast and multicast packets are sent to a pool nodethat performs this mapping. If the packet is a unicast packet, the MFEdetermines a port of a MFE that corresponds to the logical egress portof the logical forwarding element through which the packet is to be sentout (i.e., the logical port to which the intended recipient of thepacket is coupled). This information is stored in the registers for thepacket in some embodiments, for use in subsequent operations on thepacket.

In the seventh stage 1835, a different flow entry is matched. This flowentry specifies to send the packet to a different MFE, which isdetermined in the previous stage 1830, via a particular tunnel. In someembodiments, the MFE bases this decision on the physical destinationidentified in the previous stage and its flow entry which maps thisphysical destination to a particular tunnel either directly to thephysical destination or to a different MFE that can forward the packetthrough another tunnel to its physical destination.

At the eighth stage 1840, the resubmitted packet matches a flow entry tosend an observation to the controller based on (i) the trace indicatorbit stored in the register and (ii) the decision to send the packet outthrough a tunnel. As shown, the MFE sends an observation message to thenetwork controller from which it received the packet. In someembodiments, this observation message contains (i) the packet asreceived and (ii) the contents or data of the registers for the packet.

The data in the registers, which is then sent as part of observationmessage 1865, includes an (i) an indicator that the observation is aforwarding observation (i.e., that the traced packet is being sent intoa tunnel), (ii) a MFE ID specifying that the MFE 1800 is sending theobservation, and (iii) a tunnel ID denoting the tunnel through which thepacket is sent. Some embodiments also send a remote IP of the MFE atwhich the tunnel terminates, while in other embodiments the controllerdetermines this data based on its stored network state information.

As indicated, the MFE 1800 sends the packet 1855 as received to thecontroller as part of the observation message in some embodiments. Inother embodiments, the MFE sends to the controller the packet with theencapsulations that is sent to the MFE. The controller, in someembodiments, only uses the packet to read the trace data stored in thepacket (e.g., the controller ID and the trace operation session ID), andreads all other pertinent information from the register contents sent asthe observation message.

Furthermore, the MFE 1800 also sends out the packet 1855 through thetunnel to another MFE (e.g., the destination MFE, a pool node thatconnects to a destination MFE, etc.), which is now further encapsulated.In addition to the previous headers, the packet is now encapsulated witha tunnel header. In some embodiments, the MFE 1800 also clears out theregisters after processing the packet 1855 (e.g., deletes the registersfrom memory).

The previous example illustrates an MFE processing a packet from acontroller and sending a set of observation message when the packet issent to another MFE via a tunnel. In the managed networks of someembodiments, MFEs also send observations when delivering traced packetsto logical ports. FIG. 19 conceptually illustrates the processingperformed by a MFE 1900 that receives a packet from another MFE througha tunnel, processes the packet, identifies that the packet should bedelivered to a logical port from the MFE, and sends an observation tothe network controller. This figure illustrates the processing overeight stages 1905-1920.

As with the MFE 1800, the MFE 1900 of some embodiments may be a softwareforwarding element that runs on or within a hypervisor on a VM host in avirtualized environment. In this case, the MFE operates within the samemachine that hosts the destination VM for the processed packet.

As shown at the first stage 1905, the MFE 1900 includes one or moreforwarding tables 1945 and one or more registers 1930. Like theforwarding tables of the MFE 1800, the forwarding tables 1945 of someembodiments include forwarding entries pushed down to the MFE 1900 froma network controller as physical control plane data and then convertedinto physical forwarding plane data by the MFE. The forwarding tables1945, in some embodiments, include entries for several unaffiliatedlogical networks that are virtualized over the physical network.

Like the registers of the MFE 1800, the registers 1930 serve astemporary, quick-access storage for information while processing apacket. Some embodiments resubmit a packet to the forwarding tables ofthe MFE several times in order to fully process the packet. During thisprocessing, the MFE 1900 uses the registers 1850 to temporarily storeinformation (e.g., logical context information, packet tracinginformation, etc.) about the packet. In addition, some embodiments sendthe contents of the registers to the controller as part of anobservation message.

At the first stage 1905, the MFE 1900 receives a packet 1955 through atunnel from a different MFE. The packet 1955 is similar to the packet1855 after the eighth stage 1840 of FIG. 18, in that the packet isencapsulated with a logical context and tunnel information. While thetunnel through which the packet is sent terminates at two MFEs, inbetween these two MFEs may be one or more unmanaged forwarding elementsthat make up a physical network over which packets are sent. Inaddition, like the packet of the previous figure, the packet 1955includes a payload and a set of headers. As illustrated, the headersinclude a trace indicator bit that is set to “1” to indicate that thepacket is a traced packet. In some embodiments, this bit is set within alogical VLAN field of the header. While shown as a single bit, someembodiments use more than one bit in the logical VLAN field or adifferent location of the packet to indicate that a packet is traced. Inaddition, appended to the payload (either within the payload portion ofthe packet or after the payload portion), the packet 1955 includes traceoperation data.

The MFE 1900 receives the packet 1955 and begins processing the packetthrough its forwarding tables. As mentioned, the forwarding tablesinclude numerous flow entries (e.g., on the order of hundreds orthousands of entries) which take the form a match and correspondingaction. That is, if the packet matches a flow entry, the entry specifiesan action to take. As shown by the thicker line in the forwarding tables1945 in the second stage 1910, at this stage a flow entry is matchedthat specifies to modify a particular bit in a particular register inorder to indicate that the current packet is a traced packet. Thus, asshown, one of the registers 1930 is modified to specify this traceindicator bit.

In the third stage 1915, a different flow entry is matched, as shown bythe thicker line in the forwarding tables 1945. This flow entryspecifies to read the logical context of the packet as stored on thepacket as received. In this case, the packet has completed at least someof the logical forwarding operations in its logical processing pipelineat a previous MFE (e.g., the MFE at which the packet was injected) andthe logical context tag on the packet specifies the results of thesedecisions (e.g., forwarding to a particular logical port). In somecases, all of the logical forwarding operations for the packet will havebeen performed by previous MFEs, in which case the fourth and fifthstages 1920 and 1925 of this figure will be skipped. In someembodiments, many trace packets will have all of their logicaloperations performed by the first MFE at which the packets are injected.However, some embodiments perform certain operations at a later MFErather than the first hop MFE (e.g., egress ACL decisions, destinationlogical switch L2 forwarding after being sent through a logical L3router, etc.).

When at least one logical operations remains to be performed, the packetmatches a flow entry that specifies to perform the remaining set oflogical forwarding operations. In that case, the MFE will perform theseremaining logical forwarding operations and send the resultingobservations to the network controller that manages the MFE 1900, asshown by the fourth and fifth stages 1920 and 1925, which are similar tothe fourth and fifth stages 1820 and 1825 described above. As indicatedbelow, different logical forwarding or logical ACL operations may resultin different observation messages sent to the controllers.

In the sixth stage 1930, the packet matches a flow entry that specifiesto identify, based on the logical context (i.e., the logical egressport) of the packet, the physical interface of the MFE 1900 to which thelogical egress port is bound. For instance, for a destination addressthat corresponds to a VM, this is the port of the MFE to which a VIF ofthe VM connects. In the seventh stage 1935, a different flow entry ismatched, as shown by the thicker line in the forwarding tables 1945.This flow entry specifies to deliver the packet to this destination portidentified in the sixth stage. In some embodiments, the operations shownin these two stages 1930 and 1935 are in fact all specified by a singleflow entry.

At the eighth stage 1940, the resubmitted packet matches a flow entry towrite data to the registers for the packet, send an observation to thecontroller based on (i) the trace indicator bit stored in the registerand (ii) the decision made at the third stage 1915 to deliver the packetto its destination logical port, and to drop the packet (i.e., notactually deliver the packet to the destination port). As shown, the MFEsends an observation message to the network controller that manages theMFE 1900. In some embodiments, this observation message contains (i) thepacket as generated by the controller (i.e., without the logical contextand tunnel encapsulation) and (ii) the contents of the registers for thepacket. As such, in some embodiments the flow entry that specifies tosend the observation also specifies that the MFE first write certaindata into the registers. As indicated, the MFE 1900 sends the packet1955 to the controller as part of the observation message.

B. Logical Forwarding Operations

As described above, a MFE performs a set of logical forwardingoperations to forward packets in the logical networks and sendsobservations for these logical forwarding operations to the controllercluster that manages the MFE. This sub-section II.B describes differentlogical forwarding operations for which the MFE of some embodiments sendobservations to the controller.

FIG. 20 conceptually illustrates an example of a MFE that sends multipleobservations to a controller cluster while processing a trace packet.Specifically, the top half of this figure illustrates a processingpipeline 2000 that a MFE performs on a packet in order to advance thepacket in a logical forwarding element implemented across a set of MFEs.The processing pipeline 2000 includes six stages 2005-2030 forprocessing a packet.

The top half of this figure also illustrates a controller cluster 2035,a MFE 2090 that performs the processing pipeline 2000, and variouspackets and observations with encircled numbers that indicate an orderin which the packets and observations messages sent. The bottom half ofthe figure illustrates three logical forwarding elements, which are alogical switches 2045 and 2050 and a logical router 2055, which areimplemented in a physical network.

As shown, the logical switch 2045 connects two virtual machines 2060 and2065 and the logical switch 2050 connects two virtual machines 2070 and2075. The logical router 2055 routes data traffic between the twological switches. Each of the virtual machines 2060-2075 connects to alogical port of the logical switches 2045 and 2050. In some embodiments,a user (e.g., network administrator) defines the logical switches androuter, which may be part of a larger logical network. For instance, thelogical router 2055 may include a logical port that connects to anexternal gateway (e.g., to an external network), to various middleboxes,etc.

In some embodiments, the user defines the logical forwarding elementsthrough an API of a controller cluster, which translates the userdefinition into a logical control plane definition of the logicalforwarding elements. The controller cluster then converts this logicalcontrol plane definition into a logical forwarding plane specificationof the logical forwarding element. The logical forwarding planespecification, in some embodiments, includes logical forwarding tableentries (logical flow entries) that specify rules for forwarding packetsto logical ports of the logical forwarding elements. For instance, thelogical control plane of some embodiments includes bindings between MACaddresses of VMs and logical ports, and the logical forwarding planespecifies flow entries for forwarding packets to the logical ports basedon matches of the MAC addresses.

The controller cluster, as mentioned, pushes these flow entries toseveral MFEs in some embodiments, such that the logical forwardingelement (and/or other logical forwarding elements, such as logicalrouters) are implemented in a distributed, virtualized fashion. Thephysical network of FIG. 20 illustrates that the four VMs 2060-2075 arehosted on two different host machines 2080 and 2085. Some embodimentsmay put multiple VMs from a logical network on the same machine, as inthis case with the hosts 2080 and 2085. As shown, in the virtualizedenvironment, each of these hosts 2080 and 2085 also hosts additionalvirtual machines beyond those connected to the logical switches 2045 and2050. That is, many tenants may share the use of the physical network,and in fact may share use of a single physical host.

Operating on each host (e.g., within the hypervisor on the host) is aMFE. The MFE, in some embodiments, is a software forwarding element towhich the network controller connects and pushes down flow entries forvarious logical forwarding elements. In this case, because VMs from eachof the logical switches 2045 and 2050 are located on each of the twoillustrated hosts 2080 and 2085, the MFE in each of these hostsimplements both of the logical switches 2045 and 2050. That is, each ofthe illustrated MFEs has flow entries in its forwarding tables forlogically forwarding packets to the logical ports associated with thedifferent VMs 2060-2065. In addition, each of the MFEs 2090 and 2092implement the logical router 2055 in order to logically route packetssent by one of the VMs 2060-2075.

The top half of FIG. 20 will now be described. As shown by the encircled1, the controller cluster 2035 receives a request for a packet tracingoperation. In this case, the request specifies a source logical portthat maps to a VM connected to the MFE 2090 and a destination logicalport that maps to a VM connected to another MFE (e.g., MFE 2092). Inresponse to the request, the controller cluster 2035 generates a packet2002 and sends (shown by the encircled 2) this to the MFE 2090. Someembodiments send the packet to the host from a logical controller via aphysical controller, as described above by reference to, e.g., FIG. 14.The header of the packet 2002 specifies the source and destinationaddresses received in the request, as well as a trace indicator thatmarks the packet as a test packet for a trace operation.

In some embodiments, the processing pipeline 2000 may be implemented byflow entries in the MFEs. These flow entries process the packet bymatching various fields either stored in the packet header (e.g., sourceand destination MAC addresses) or stored in registers for the packetcreated at the MFE (e.g., physical and logical ingress ports, logicalforwarding element IDs, etc.). As mentioned, in some embodiments theseflow entries are pushed down by the controller cluster as physicalcontrol plane data and converted into the physical forwarding plane datathat makes up the forwarding tables.

When the MFE 2090 receives the packet 2002, the MFE 2090 stores aregister bit indicating that the packet is marked for a trace operation(either based on a command sent with the packet by the controllercluster 2035 or a flow entry that matches the trace indicator bit in thepacket). The MFE 2090 then performs an ingress context mapping on thepacket to determine the ingress context of the packet in the first stage2005 of the processing pipeline 2000. In some embodiments, thecontroller cluster sets register bits for the packet at the MFE thatindicate a physical ingress port in order to mimic the packet havingactually been received through that physical ingress port of the MFE(e.g., the physical ingress port to which the VM 2060 connects). Theingress context mapping of some embodiments maps this physical ingressport to a logical ingress port of a logical switching element (e.g.,logical switch 2045).

In the second stage 2010 of the processing pipeline 2000, a set oflogical operations are performed on the packets by the flow entries thatimplement the logical switch 2045 at the MFE 2090. These operations areperformed first because the ingress context mapping mapped the sourcephysical ingress port to a logical port of the logical switch 2045. Theoperations that implement the logical switch 2045, in some embodiments,include at least a logical L2 forwarding operation that identifies adestination logical port on the logical switch 2045 based on thedestination address stored in the packet header (e.g., the port thatconnects to the logical router 2055). The processing 2010 mayadditionally include logical L2 ingress and/or egress ACL decisions,respectively performed before and after the logical L2 forwardingoperation. These L2 operations will be described in further detail belowby reference to FIG. 21.

In some embodiments, the MFE 2090 sends (shown by the encircled 3) anobservation for each of the L2 operations for the logical switch 2045.In some embodiments, the MFE 2090 is configured (by the controllercluster 2035) to send an observation to the controller cluster afterperforming each L2 operation for the logical switch 2045 when theregister bit for the packet indicates that the packet is marked for thetrace operation. Because the packet 2002 is marked for the tracingoperation, the MFE 2090 of some embodiments sends an observationincluding the result of performing an L2 operation for the logicalswitch 2045. In some embodiments, the MFE 2090 is configured to sendobservations only for some of the L2 operations for the logical switch2045 performed on the marked packet (e.g., only certain ACL orforwarding operations).

In this example, the packet is logically sent to the logical router 2055through a logical port of the logical switch 2045 because thedestination MAC address of the packet corresponds to a logical port ofthe logical router, which is linked to the logical switch 2045. Becausethe packet is sent from the logical switch 2045 to the logical router2055, both of which are implemented within the MFE 2090, the packet doesnot physically leave the MFE 2090. Instead, the packet is resubmitted tothe MFE 2090 so that the MFE can perform the logical L3 routeroperations.

In the third stage 2015 of the processing pipeline 2000, a set oflogical forwarding operations are performed on the packets by the flowentries that implement the logical router 2055 at the MFE 2090. Theoperations that implement the logical router 2055, in some embodiments,include at least a logical L3 forwarding operation that identifies adestination logical port on the logical router 2055 (e.g., based on thedestination address stored in the packet header). In this case, thisdestination logical port is the port that connects to the logical switch2050. The processing 2015 may additionally include logical L3 ingressand/or egress ACL decisions, respectively performed before and after thelogical L3 forwarding operation. These L3 operations will be describedin further detail below by reference to FIG. 21.

In some embodiments, the MFE 2090 sends (shown by the encircled 4) anobservation for each of the L3 operations of the logical switch 2045. Insome embodiments, the MFE 2090 is configured (by the controller cluster2035) to send an observation to the controller cluster after performingeach L3 operation for the logical router 2055 when the register bit forthe packet indicates that the packet is marked for the tracingoperation. Because the packet 2002 is marked for the tracing operation,the MFE 2090 of some embodiments sends an observation including theresult of performing each L3 operation for the logical router 2055. Insome embodiments, the MFE 2090 is configured to send observations onlyfor some of the L3 operations that the logical router 2055 performed onthe marked packet (e.g., only certain ACL or forwarding operations).

In this example, the logical router 2055 routes the packet to thelogical switch 2050 because the destination IP address of the packetmaps to the output port (e.g., by performing routing functions such asDNAT, identifying a longest prefix match, etc.) of the logical routerthat connects to the logical switch 2050. Because the packet is sentfrom the logical router 2055 to the logical switch 2050, which is alsoimplemented within the MFE 2090, the packet does not leave the MFE 2090within the physical network. Instead, the packet is resubmitted to theMFE 2090 so that the MFE can perform the logical L2 switching operationsfor the second logical switch 2050.

In the fourth stage 2020 of the processing pipeline 2000, an additionalset of logical forwarding operations are performed on the packets by theflow entries that implement the logical switch 2050 at the MFE 2090. Theoperations that implement the logical switch 2055, in some embodiments,include at least a logical L2 forwarding operation that identifies adestination logical port on the logical switch 2050 (e.g., based on thedestination address stored in the packet header). In this case, thedestination logical port is the port that connects to the VM 2075, theIP address of which is the destination address of the trace packet 2002.The processing 2020 may additionally include logical L2 ingress and/oregress ACL decisions in some embodiments. In other embodiments, theegress ACL decisions are performed by the destination MFE 2092 ratherthan the first hop MFE 2090.

In some embodiments, the MFE 2090 sends (shown by the encircled 5) anobservation for each of the L2 operations of the logical switch 2050. Insome embodiments, the MFE 2090 is configured (by the controller cluster2035) to send an observation to the controller cluster after performingeach L2 operation for the logical switch 2050 when the register bit forthe packet indicates that the packet is marked for the trace operation.Because the packet 2002 is marked for the tracing operation, the MFE2090 of some embodiments sends an observation including the result ofperforming each L2 operation for the logical switch 2050. In someembodiments, the MFE 2090 is configured to send observations only forsome of the L2 operations for the logical switch 2050 performed on themarked packet (e.g., only certain ACL or forwarding operations).

Next, the fifth stage 2030 of the processing pipeline 2000 is similar tothe third stage 430 of the processing pipeline 400 described above byreference to FIG. 4. That is, the fifth stage 2030 of the processingpipeline 2000 performs a mapping operation on the packet to map thelogical egress port (e.g., the port of logical switch 2050 that connectsto the destination VM 2075) to a physical destination. That is, themapping operation determines one or more ports of one or more MFEs thatcorrespond to the logical egress port of the logical switch throughwhich the packet is to be sent out.

At the sixth stage 2025 of the processing pipeline 2000, a set ofoperations to implement physical forwarding of the packet is performed.The set of physical operations of some embodiments includes operationsfor sending the packet to the physical port(s) that correspond to thelogical egress port(s) of the packet, as determined in the previousstage 2025 of the pipeline. For example, the set of physical operationsof some embodiments determines one or more ports of the MFE on which theprocessing pipeline 2000 is performed through which to send the packetout in order for the packet to reach the physical port(s) determined atthe fifth stage 2025. This way, the MFEs can route the packet along thecorrect path in the network for the packet to reach the determinedphysical port(s) that correspond to the packet destination.

In some embodiments, the MFE 2090 sends (shown by the encircled 6) oneor more observation messages 2065 to the controller cluster for the setof physical operations performed at the fourth stage 2040 when theregister bit indicates that the packet is marked for the traceoperation. For instance, the MFE of some embodiments sends observationswhen either of two actions is taken by the MFE: (1) the MFE sends thepacket to another MFE via a tunnel, or (2) the MFE delivers the packetto a physical port to which the logical egress port is mapped. In thiscase, if sending the packet through a tunnel to the MFE 2092 fordelivery to the VM 2075, the MFE 2090 would send a forwardingobservation message. On the other hand, if the MFE delivered the packetto the VM 2070 (or, would deliver the packet if it were not a traceoperation packet), the MFE would send a delivery observation message.

As mentioned above, in some embodiments, the processing pipeline 2000 isperformed by each MFE in the managed network that is used to implementthe logical forwarding element. In some embodiments, some of the MFEsperform only a portion of the processing pipeline 2000. For example, insome embodiments, the MFE that initially receives the packet may performthe first-sixth stages 2005-2030 and the remaining MFEs thatsubsequently receive the packet only perform the first, fifth, and sixthstages 2005, 2025, and 2030.

In the example of FIG. 20, the logical forwarding operations for all ofthe logical forwarding elements—the L2 processing operations for thelogical switch 2045, the L3 processing operations for the logical router2055, and the L2 processing operations for the logical switch 2050—areperformed by the MFE 2090. In some embodiments, the controller clusterconfigures the MFEs in such a way that the logical forwarding operationsfor a packet are performed by the MFEs in a distributed manner. Forinstance, the MFEs can be configured such that the MFE 2090 performs theL2 processing operations for the logical switch 2045 and the L3processing operations for the logical router 2055 and another MFEperforms the L2 processing operations for the logical switch 2050. Anexample of performing logical forwarding operations in a distributedmanner by the MFEs is described below by reference to FIG. 22. However,before FIG. 22 illustrates the example, FIG. 21 illustrates L2 and L3operations in detail.

FIG. 21 conceptually illustrates an example of more detailed processingpipeline 2000 described above by reference to FIG. 20. Specifically,FIG. 21 illustrates the L2 processing operations at the stage 2010, theL3 processing operations at the stage 2015, and the L2 processingoperations at the stage 2020 of FIG. 20 in more details. FIG. 21 alsoillustrates the MFE 2090 and the packet 2002.

In some embodiments, each operation in a processing pipeline performedby a MFE is based on one or more flow entries stored in the forwardingtables of the MFE. In order to configure the MFE to send an observationfor an operation, the controller cluster of some embodiments populatesthe forwarding tables with such flow entries that direct the MFE to sendan observation at the time or after the operation is performed. In someembodiments, one of these flow entries will be matched when the packetis resubmitted to the MFE after the MFE performs the operation. In otherembodiments, a single matched flow entry specifies to both perform thelogical processing pipeline operation and send the correspondingobservation to the controller cluster.

Some embodiments send an observation to the controller cluster only fora subset of logical or physical forwarding operations that a MFEperforms on a packet. For instance, the MFE 2090 is configured to sendan observation after performing an operation depicted as a grey boxillustrated in this figure. Therefore, when the MFE 2090 resubmits thepacket 2002 after performing an operation depicted as a grey box, a flowentry is matched that specifies to send an observation to the controllercluster (e.g., the physical controller that manages the MFE 2090). Asmentioned, in other embodiments, the flow entry for performing theseoperations also specifies to send the observation, rather than havingthe operation and the observation message as two separate flow entries.Each curvy, hollow-headed arrow on top of grey boxes indicates that theobservation for the operations depicted as the grey box is sent to thecontroller cluster.

The operations depicted as boxes in this figure will now be described.The ingress context mapping stage 2005 is described above, and in someembodiments does not result in an observation message sent to thecontroller cluster. For the next stage 2010, the L2 processingoperations at this stage include three operations at sub-stages2105-2115 for processing a packet through the logical switch 2045. Atthe stage 2105, the MFE 2090 performs a logical L2 ingress ACL operationbased on the packet's logical context to determine ingress accesscontrol of the packet with respect to the logical switch 2045. Forinstance, an ingress ACL is applied to the packet to control thepacket's access to the logical switch 2045 when the logical switch 2045receives the packet. Based on the ingress ACL defined for the logicalswitch 2045, the packet may be allowed into the logical switch 2045 orthe packet may be dropped. For instance, some embodiments ensure thatthe packet's source MAC address is correct for the logical port fromwhich it was received.

The observation message of some embodiments, as with the previouslydescribed observations, includes both the packet as received at the MFEand data copied to the registers for the packet for the purpose of beingsent in the observation message. This data includes, for a L2 ingressACL decision in some embodiments, (i) a MFE ID specifying that the MFE2090 is sending the observation, (ii) a logical switch ID for the switch2045 that implements the ACL conditions (and, in some embodiments, thelogical ingress port on that logical switch), (iii) the matching ACLentry (e.g., a port security entry, QoS entry, etc.), and (iv) thedecision of the ACL entry (e.g., whether to drop, allow, enqueue, etc.the packet). One of ordinary skill in the art will recognize thatdifferent logical ingress ACL observation messages may include differentinformation.

At the following stage 2110, an L2 forwarding operation is performed onthe packet. In some embodiments, the MFE 2090 performs this L2forwarding operation based on the packet's destination MAC address inorder to send the packet through a logical egress port of the logicalswitch 2045 (i.e., the logical port that corresponds to the destinationaddress of the packet, which at this point in this case is the addressof the logical router).

The observation message of some embodiments again includes both thepacket as received at the MFE (or as modified to include the logicalegress port, in other embodiments) and data copied to the registers forthe packet for the purpose of being sent in the observation message.This data includes, for a logical L2 forwarding decision in someembodiments, (i) a MFE ID specifying that the MFE 2090 is sending theobservation, (ii) a logical switch ID for the switch 2045 through whichthe packet is processed, and (iii) the logical egress port of thelogical switch to which the packet is sent. One of ordinary skill in theart will recognize that different logical L2 forwarding observationmessages may include different information.

Next, at the stage 2115, the MFE 2090 performs a L2 egress ACL operationbased on the packet's logical egress context to determine egress accesscontrol of the packet with respect to the logical switch 2045. Forinstance, an egress ACL may be applied to the packet to control thepacket's access out of the logical switch 2045. Based on the egress ACLdefined for the logical switch 2045, the packet may be dropped. Forexample, some embodiments ensure that the packet's source is allowed tosend packets to the packet's destination.

The observation message of some embodiments again includes both thepacket as received at the MFE and data copied to the registers for thepacket for the purpose of being sent in the observation message. Thisdata includes, for a L2 egress ACL decision in some embodiments, (i) aMFE ID specifying that the MFE 2090 is sending the observation, (ii) alogical switch ID for the switch 2045 that implements the ACL conditions(and, in some embodiments, the logical egress port on that logicalswitch), (iii) the matching ACL entry (e.g., a port isolation entry,etc.), and (iv) the decision of the ACL entry (e.g., whether to drop,allow, enqueue, etc. the packet). One of ordinary skill in the art willrecognize that different logical egress ACL observation messages mayinclude different information.

When the packet's destination address is not connected to a logical portof the logical switch 2045, the logical switch 2045 sends the packet tothe logical router 2055. The MFE then performs the L3 processingoperations at stage 2015 in order to route the packet to thedestination. The L3 processing operations in some embodiments includefive operations at sub-stages 2120-2140 for processing the packet 2002through the logical router 2055 (shown in FIG. 20) that is implementedby the MFE 2090. In some embodiments, L3 processing operations involveperforming a set of logical routing operations for determining how toroute the packet through a logical layer 3 network.

At the stage 2120, the MFE 2090 performs a L3 ingress ACL operation fordetermining access control when the logical router 2055 receives thepacket 2002. In some embodiments, the MFE 2090 performs the L3 ingressACL operation based on the packet's logical ingress context to determineingress access control of the packet with respect to the logical router2055. For instance, some embodiments determine whether packets from thesource of the packet are allowed through the logical ingress port of thelogical router.

The observation message of some embodiments again includes both thepacket as received at the MFE and data copied to the registers for thepacket for the purpose of being sent in the observation message. Thisdata includes, for a L3 ingress ACL decision in some embodiments, (i) aMFE ID specifying that the MFE 2090 is sending the observation, (ii) alogical router ID for the router 2055 that implements the ACL conditions(and, in some embodiments, the logical ingress port on that logicalrouter), (iii) the matching ACL entry (e.g., a port security entry, QoSentry, etc.), and (iv) the decision of the ACL entry (e.g., whether todrop, allow, enqueue, etc. the packet). One of ordinary skill in the artwill recognize that different logical ingress ACL observation messagesmay include different information.

At the next stage 2125, the MFE 2090 of some embodiments performs adestination network address translation (DNAT) to revert the destinationaddress of the packet back to the real address of the destinationmachine that is hidden from the source machine of the packet. This stage2136 is performed when DNAT is implemented by the MFE 2090. In otherembodiments, DNAT may not be necessary or may not be implemented withinthe particular logical network.

The observation message of some embodiments again includes both thepacket as received at the MFE and data copied to the registers for thepacket for the purpose of being sent in the observation message. Thisdata includes, for a DNAT operation in some embodiments, (i) a MFE IDspecifying that the MFE 2090 is sending the observation, (ii) a logicalrouter ID for the router 2055 within which the DNAT processing isimplemented, (iii) the original destination IP address, and (iv) thetranslated destination IP address. One of ordinary skill in the art willrecognize that different DNAT observation messages may include differentinformation. Furthermore, some embodiments do not include observationmessages for DNAT performed within the processing pipeline, or may evenperform the DNAT using a middlebox located outside of the processingpipeline.

At the stage 2130, the MFE 2090 performs a logical L3 routing operationto determine one or more logical egress ports of the logical routerbased on the L3 address (e.g., destination IP address) of the packet.The observation message of some embodiments, as with the previouslydescribed observations, includes both the packet as received at the MFEand data copied to the registers for the packet for the purpose of beingsent in the observation message. This data includes, for a logical L3routing decision in some embodiments, (i) a MFE ID specifying that theMFE 2090 is sending the observation, (ii) a logical router ID for therouter 2055 through which the packet is processed, (iii) the routingentry matched to forward the packet, and (iv) the logical egress port ofthe logical router to which the packet is sent. One of ordinary skill inthe art will recognize that different logical L3 forwarding observationmessages may include different information.

At the next stage 2135, the MFE 2090 performs source network addresstranslation (SNAT) on the packet if SNAT is implemented by the MFE 2090.By performing an SNAT operation on the packet, the MFE 2090 replaces thesource IP address of the packet with a different IP address in order tohide the real source IP address. Like the DNAT operation 2125, in otherembodiments, the SNAT may not be implemented within the particularlogical network.

The observation message of some embodiments again includes both thepacket as received at the MFE and data copied to the registers for thepacket for the purpose of being sent in the observation message. Thisdata includes, for a SNAT operation in some embodiments, (i) a MFE IDspecifying that the MFE 2090 is sending the observation, (ii) a logicalrouter ID for the router 2055 within which the SNAT processing isimplemented, (iii) the original source IP address, and (iv) thetranslated source IP address. One of ordinary skill in the art willrecognize that different SNAT observation messages may include differentinformation. Furthermore, some embodiments do not include observationmessages for SNAT performed within the processing pipeline, or may evenperform the SNAT using a middlebox located outside of the processingpipeline.

In some embodiments, the L3 processing additionally includes performinga destination MAC address replacement operation, which may includesending Address Resolution Protocol (ARP) requests. The MFE sends ARPrequest packets to all the VMs on the logical L2 domain identified ascorresponding to the destination port of the logical router. The MFEshould receive an ARP reply packet from one of these VMs with a MACaddress, which it uses to replace the destination MAC address in thepacket before sending the packet to the second logical switch. Someembodiments trace these ARP packets and the processing thereof as wellas the original packet, while other embodiments do not also trace theARP request and reply packets.

At the stage 2135, the MFE 2090 performs a L3 egress ACL operation fordetermining access control before the logical router 2055 routes thepacket out of the logical router 2055 through the logical portdetermined in the stage 2130. The L3 egress ACL operation is performedbased on the L3 addresses (e.g., source and destination IP addresses) ofthe packet in some embodiments. For instance, some embodiments ensurethat packets from the source IP address are allowed to exit the logicalrouter through the logical egress port identified at the stage 2130.

The observation message of some embodiments again includes both thepacket as received at the MFE and data copied to the registers for thepacket for the purpose of being sent in the observation message. Thisdata includes, for a L3 egress ACL decision in some embodiments, (i) aMFE ID specifying that the MFE 2090 is sending the observation, (ii) alogical router ID for the router 2055 that implements the ACL conditions(and, in some embodiments, the logical ingress port on that logicalrouter), (iii) the matching ACL entry (e.g., a port isolation entry,etc.), and (iv) the decision of the ACL entry (e.g., whether to drop,allow, enqueue, etc. the packet). One of ordinary skill in the art willrecognize that different logical egress ACL observation messages mayinclude different information.

For the next stage 2020, the L2 processing operations at this stageinclude three operations at sub-stages 2145-2155 for processing a packetthrough the logical switch 2050 shown in FIG. 20. These stages aresimilar to the stages 2105-2115 for the first logical switch throughwhich the packet is processed.

At the stage 2145, the MFE 2090 performs an L2 ingress ACL operationbased on the packet's logical context to determine ingress accesscontrol of the packet with respect to the logical switch 2050. Forinstance, an ingress ACL is applied to the packet to control thepacket's access to the logical switch 2050 when the logical switch 2050receives the packet. Based on the ingress ACL defined for the logicalswitch 2050, the packet may be allowed into the logical switch 2050 orthe packet may be dropped. For instance, some embodiments ensure thatpackets from the source IP address stored in the packet header areallowed to enter through the logical ingress port of the logical switch2050.

The observation message of some embodiments, as with the previouslydescribed L2 ingress ACL observation from stage 2105, includes both thepacket as received at the MFE and data copied to the registers for thepacket for the purpose of being sent in the observation message. Thisdata includes, for a L2 ingress ACL decision in some embodiments, (i) aMFE ID specifying that the MFE 2090 is sending the observation, (ii) alogical switch ID for the switch 2050 that implements the ACL conditions(and, in some embodiments, the logical ingress port on that logicalswitch), (iii) the matching ACL entry (e.g., a port security entry, QoSentry, etc.), and (iv) the decision of the ACL entry (e.g., whether todrop, allow, enqueue, etc. the packet). One of ordinary skill in the artwill recognize that different logical ingress ACL observation messagesmay include different information.

At the stage 2150, an L2 forwarding operation is performed on thepacket. In some embodiments, the MFE 2090 performs this L2 forwardingoperation based on the packet's destination address in order to send thepacket through a logical egress port of the logical switch 2050 (i.e.,the logical egress port that corresponds to the destination MAC addressof the packet).

The observation message of some embodiments, as with the previouslydescribed L2 forwarding observation from stage 2110, includes both thepacket as received at the MFE (or as modified to include the logicalegress port, in other embodiments) and data copied to the registers forthe packet for the purpose of being sent in the observation message.This data includes, for a logical L2 forwarding decision in someembodiments, (i) a MFE ID specifying that the MFE 2090 is sending theobservation, (ii) a logical switch ID for the switch 2050 through whichthe packet is processed, and (iii) the logical egress port of thelogical switch to which the packet is sent. One of ordinary skill in theart will recognize that different logical L2 forwarding observationmessages may include different information.

Next, at the stage 2155, the MFE 2090 performs a L2 egress ACL operationbased on the packet's logical egress context to determine egress accesscontrol of the packet with respect to the logical switch 2050. Forinstance, an egress ACL may be applied to the packet to control thepacket's access out of the logical switch 2050. Based on the egress ACLdefined for the logical switch 2050, the packet may be dropped. Forexample, some embodiments ensure that the packet's source is allowed tosend packets to the packet's destination logical port on the switch2050.

The observation message of some embodiments again includes both thepacket as received at the MFE and data copied to the registers for thepacket for the purpose of being sent in the observation message. Thisdata includes, for a L2 egress ACL decision in some embodiments, (i) aMFE ID specifying that the MFE 2090 is sending the observation, (ii) alogical switch ID for the switch 2050 that implements the ACL conditions(and, in some embodiments, the logical egress port on that logicalswitch), (iii) the matching ACL entry (e.g., a port isolation entry,etc.), and (iv) the decision of the ACL entry (e.g., whether to drop,allow, enqueue, etc. the packet). One of ordinary skill in the art willrecognize that different logical egress ACL observation messages mayinclude different information.

The stages 2025 and 2030 are described above. These stages map thelogical output port of the second logical switch 2050 to a physicaldestination, then perform a physical forwarding operation on the packetto either send the packet through a tunnel to another MFE or to deliverthe packet to its destination (e.g., a VM, a gateway to an externalnetwork, etc.). As described above, either of these physical operations(forwarding into a tunnel, delivery) result in observations sent to thecontroller cluster.

In the previous example, all of the logical processing is performed bythe first managed forwarding element that receives the traced packetfrom the controller cluster. In this situation, only this MFE will sendobservation messages regarding logical processing to the controllercluster, while other MFEs may send physical forwarding or deliveryobservation messages, as described in section I above. In someembodiments, the last MFE performs the L2 egress ACL for the secondlogical switch, in which case this MFE may send observation messagesdetailing the logical ACL decisions.

However, in some embodiments, the controller cluster configures the MFEsto perform a set of logical processing operations in a more distributedmanner. FIG. 22 conceptually illustrates an example of performinglogical processing operations in such a distributed manner by the MFEs.Specifically, this figure illustrates the two MFEs 2090 and 2092 thatimplement the logical switches 2045 and 2050 and the logical router 2055described above by reference to FIG. 20. FIG. 22 also illustrates alogical controller 2205 and two physical controllers 2210 and 2215,which make up at least a portion of the controller cluster 2035 in thisexample. This figure illustrates various packets and observations withencircled numbers that indicate an order in which the packets andobservations messages sent within the network. However, when a pathdiverts into branches (e.g., 8 and 10) the numbers may not necessarilyrepresent the order in different branches, owing to different lag timesfor different operations.

As shown by the encircled 1, the logical controller 2205 receives arequest from the user. This request specifies a source logical port thatmaps to a VM connected to the MFE 2090. In response to the request, thelogical controller 2205 generates a packet 2092, as described above. Asshown by the encircled 2, the logical controller then sends thegenerated packet to the physical controller 2210 because the physicalcontroller 2210 manages the MFE 2090. The header of the packet 2092specifies the source and destination addresses received in the request,as well as a trace indicator that marks the packet as a test packet fora trace operation.

The physical controller 2210 examines the packet and identifies that theMFE 2090 is the MFE into which to inject the packet because the logicalport to which the source address of the packet connects is mapped to aphysical port of the MFE 2090. As shown by the encircled 3, the physicalcontroller 2210 sends the packet 2002 to the MFE 2090. In addition, aspreviously described, the physical controller 2210 may also modifyregister bits for the packet at the MFE in order to simulate thereceiving of the packet through the appropriate physical port of theMFE.

Upon receiving the packet, the MFE 2090 starts to perform a processingpipeline 2200. As shown, the processing pipeline 2200 performed by theMFE 2090 includes five stages 2235-2240. The first three stages2235-2245 are similar to the stages 2005-2015 described above byreference to FIGS. 20 and 21. That is, at the stage 2235, the MFE 2090performs logical ingress context mapping to identify a logical ingressport of a particular logical switch based on the physical port throughwhich the packet was received (or, in this case, through which the MFEthinks the packet was received based on the modified register for thepacket).

The MFE 2090 then performs L2 processing operations 2240 and L3processing operations 2245, which result in the MFE sending observationsto the physical controller (as shown by the encircled 4 and 6), and thephysical controller sending these observations or the analyses thereofto the logical controller (as shown by the encircled 5 and 7). However,the MFE 2090 in this example is not configured (by the controllers) toperform L2 processing operations for the logical switch 2050. Instead,the MFE 2092 is configured to perform the L2 processing operations forthe logical switch 2050. Therefore, when the MFE 2090 resubmits thepacket for further processing after the L3 forwarding has beencompleted, the packet matches a flow entry for the logical to physicalmapping operation at the stage 2250. Based on the logical L3 routingdecision made at stage 2245, the MFE maps the logical egress port of thelogical router to a physical destination (indicating to send the packetencapsulated with this egress port, that connects to the logical switch2050, to the MFE 2092).

At the stage 2255 of the processing pipeline 2200, the MFE performs aset of physical operations. The set of physical operations in thisexample determines operations for sending the packet to the MFE 2092identified in the previous stage 2250 as the MFE to which to send thepacket. Specifically, the set of physical operations determines a portof the MFE 2090 from which there is a tunnel to the MFE 2092 (or to apool node when the MFE 2090 does not have a tunnel established to theMFE 2092). While the MFE 2090 performs the operation at this stage 2255,the MFE sends the observations to the physical controller 2210 as shownby the encircled 8. The physical controller 2210, in turn, sends theobservations or the analyses of the observations to the logicalcontroller 2205 as shown by the encircled 9. The MFE 2092 also sends thepacket through the tunnel as shown by the encircled 10.

When the MFE 2092 receives the packet, the MFE 2092 performs theprocessing pipeline 2201, which is the remainder of the logicalprocessing pipeline for the packet. As shown, the processing pipeline2201 includes four stages 2260-2275. At the stage 2260, the MFEidentifies the logical context of the packet 2002. As mentioned above,the logical context would indicate that L2 processing operations for thelogical switch 2045 and the L3 processing operations for the logicalrouter 2055 have already been performed on the packet. That is, thelogical context would indicate that L2 processing operations for thelogical switch 2050 are still left to be performed. In some embodiments,the logical context simply indicates either a logical egress port of thelogical router 2055 or a logical ingress port of the logical switch2050.

Thus, when the MFE resubmits the packet, the flow entries that specifyto perform L2 processing operations for the logical switch 2050 arematched and the MFE accordingly performs the L2 processing operationsfor the logical switch 2050 at the stage 2265. By performing the L2processing operations at the stage 2260, the MFE 2092 determines alogical port of the logical switch 2050 through which the packet shouldexit. While the MFE 2092 performs the operation at this stage 2260, theMFE sends the observations to the physical controller 2215 as shown bythe encircled 11. The physical controller 2215, in turn, sends theobservations or the analyses of the observations to the logicalcontroller 2205 as shown by the encircled 12.

Next, at the stage 2270, the MFE performs the logical to physicalmapping operation to identify the physical port to which the logicalport identified in the previous stage 2265 is mapped. At the stage 2275,the MFE performs a set of physical forwarding operations and finds outthat the physical port identified in the previous stage 2270 is one ofthose located at the MFE 2092. Therefore, the MFE 2092 sends (as shownby the encircled 13) observation indicating the delivery of the packetbut drops the packet so that the packet does not reach the destinationaddress. The physical controller 2215 receives these observations fromthe MFE 2092 because the physical controller 2215 manages the MFE 2092.The physical controller 2215 sends (as shown by the encircled 14) theseobservations to the logical controller 2205. The logical controller 2205generates a report of this trace operation and sends this report to theuser as shown by the encircled 15.

The example of FIG. 22 illustrates one possible distribution of logicalforwarding operations. However, there may be numerous other possibledistributions of logical forwarding operations. Some of these possibledistributions are described in U.S. Publication 2013/0044636, which isincorporated by reference above.

In addition to the distributed processing pipeline, some embodiments mayuse a centralized router for performing the logical L3 processing, asdescribed in U.S. Publication 2013/0044636, incorporated by referenceabove. In some of these embodiments, the routing is performed by anamespace, which is a container (e.g., in a linux kernel) created toimplement a logical router (e.g., in a second level MFE such as a poolnode or a gateway), perform network address translation, etc. Inaddition, some embodiments require forwarding packets to middleboxes,which may be implemented as either centralized appliances (e.g., in aphysical box, in a separately running virtual machine) or as distributedapplications running alongside the edge MFEs.

In either of these situations, some embodiments mark the packets in sucha way that the trace indicator bit is not lost upon sending the packetto a centralized logical router or middlebox and receiving the packetback from this entity. Without a specific manner of marking the packet,the trace indicator could be lost, and no observations would be sentonce the packet is received back from the centralized entity. Forexample, if the middlebox appliance or other entity uses a tunnelingprotocol to send/receive packets (e.g., GRE, STT, etc.), then the MFEcan be configured to incorporate the marking information (which can bereferred to as a persistence register) into the tunnel header in someembodiments. For a logical router implemented in a namespace, someembodiments mark the packet in such a way that the MFE writes the bitinto a data structure used by the namespace to represent the packet(e.g., by using a specific mark field understood by the namespace). As aspecific example, some embodiments use a “mark” field of a “sk_buff”data structure in the linux kernel. The namespace will then pass thepacket back with this mark upon completing its logical processing.

IV. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 23 conceptually illustrates an electronic system 2300 with whichsome embodiments of the invention are implemented. The electronic system2300 can be used to execute any of the control, virtualization, oroperating system applications described above. The electronic system2300 may be a computer (e.g., a desktop computer, personal computer,tablet computer, server computer, mainframe, a blade computer etc.),phone, PDA, or any other sort of electronic device. Such an electronicsystem includes various types of computer readable media and interfacesfor various other types of computer readable media. Electronic system2300 includes a bus 2305, processing unit(s) 2310, a system memory 2325,a read-only memory 2330, a permanent storage device 2335, input devices2340, and output devices 2345.

The bus 2305 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 2300. For instance, the bus 2305 communicativelyconnects the processing unit(s) 2310 with the read-only memory 2330, thesystem memory 2325, and the permanent storage device 2335.

From these various memory units, the processing unit(s) 2310 retrieveinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments.

The read-only-memory (ROM) 2330 stores static data and instructions thatare needed by the processing unit(s) 2310 and other modules of theelectronic system. The permanent storage device 2335, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system2300 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 2335.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, etc.) as the permanent storage device. Like the permanentstorage device 2335, the system memory 2325 is a read-and-write memorydevice. However, unlike storage device 2335, the system memory is avolatile read-and-write memory, such a random access memory. The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the invention's processes are stored inthe system memory 2325, the permanent storage device 2335, and/or theread-only memory 2330. From these various memory units, the processingunit(s) 2310 retrieve instructions to execute and data to process inorder to execute the processes of some embodiments.

The bus 2305 also connects to the input and output devices 2340 and2345. The input devices enable the user to communicate information andselect commands to the electronic system. The input devices 2340 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 2345 display images generated by theelectronic system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 23, bus 2305 also couples electronic system2300 to a network 2365 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 2300 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”,“processor”, and “memory” all refer to electronic or other technologicaldevices. These terms exclude people or groups of people. For thepurposes of the specification, the terms display or displaying meansdisplaying on an electronic device. As used in this specification, theterms “computer readable medium,” “computer readable media,” and“machine readable medium” are entirely restricted to tangible, physicalobjects that store information in a form that is readable by a computer.These terms exclude any wireless signals, wired download signals, andany other ephemeral signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figures(including FIGS. 6, 12, and 13) conceptually illustrate processes. Thespecific operations of these processes may not be performed in the exactorder shown and described. The specific operations may not be performedin one continuous series of operations, and different specificoperations may be performed in different embodiments. Furthermore, theprocess could be implemented using several sub-processes, or as part ofa larger macro process.

We claim:
 1. For a first network controller that manages a set oflogical forwarding elements implemented in a plurality of managedforwarding elements, a method comprising: receiving, at the firstnetwork controller, a request to trace a specified packet having aparticular source on a logical forwarding element; generating thespecified packet according to the request, the generated packetcomprising an indicator that the packet is for a trace operation;sending the generated packet to a second network controller that managesa managed forwarding element associated with the particular source; andreceiving, from a set of network controllers, a first set of messagesregarding operations performed on the packet, wherein the set of networkcontrollers receives a second set of messages regarding operationsperformed on the packet from a set of managed forwarding elements thatprocess the packet.
 2. The method of claim 1, wherein the first networkcontroller does not directly communicate with any managed forwardingelement.
 3. The method of claim 1, wherein the first set of messages andthe second set of messages are identical.
 4. The method of claim 1,wherein the set of network controllers generates the first set ofmessages by analyzing the second set of messages.
 5. The method of claim1, wherein the set of network controllers includes the second networkcontroller.
 6. The method of claim 1 further comprising receiving athird set of messages regarding operations performed on the packet froma third network controller that receives the third set of messages froma second set of network controllers, wherein the particular set ofnetwork controllers receives a fourth set of messages from a second setof managed forwarding elements that process the packet.
 7. The method ofclaim 1, wherein the second network controller receives other packetsfor trace operations from other network controllers that receiverequests to trace the other packets.
 8. The method of claim 1 furthercomprising: generating a report of the trace operation based on dataregarding the packet in the first set of messages; and sending thereport to a user that requested the trace operation.
 9. The method ofclaim 1, wherein the managed forwarding element associated with theparticular source sends a plurality of messages to the second networkcontroller.
 10. A machine readable medium storing a program forexecution by at least one processing unit of a first network controllerthat manages a set of logical forwarding elements implemented in aplurality of managed forwarding elements, the program comprising sets ofinstructions for: receiving a request to trace a specified packet havinga particular source on a logical forwarding element; generating thespecified packet according to the request, the generated packetcomprising an indicator that the packet is for a trace operation;sending the generated packet to a second network controller that managesa managed forwarding element associated with the particular source; andreceiving, from a set of network controllers, a first set of messagesregarding operations performed on the packet, wherein the set of networkcontrollers receives a second set of messages regarding operationsperformed on the packet from a set of managed forwarding elements thatprocess the packet.
 11. The machine readable medium of claim 10, whereinthe particular source is a logical port of the logical forwardingelement.
 12. The machine readable medium of claim 11, wherein thelogical port is associated with a MAC address of a virtual interface ofa virtual machine that connects to the logical forwarding element. 13.The machine readable medium of claim 10, wherein the request is receivedfrom a third network controller that generates the request based on userinputs.
 14. For a first network controller that manages a firstplurality of managed forwarding elements, a method comprising:receiving, at the first network controller, a specified packet having(i) a particular source on a logical forwarding element and (ii) anindicator that the packet is for a trace operation from a second networkcontroller; inserting the received packet into a managed forwardingelement associated with the particular source such that the managedforwarding element processes the packet as though the packet wasreceived from the particular source; receiving, at the first networkcontroller, a first set of messages from a first set of managedforwarding elements in the first plurality of managed forwardingelements that processes the packet regarding operations performed on thepacket; and sending the received first set of messages to the secondnetwork controller, wherein the second network controller receives asecond set of messages from a third network controller that receives thesecond set of messages from a second set of managed forwarding elementsin a second plurality of managed forwarding elements that processes thepacket, wherein the third network controller manages the secondplurality of managed forwarding elements.
 15. The method of claim 14,wherein inserting the packet into the managed forwarding elementcomprises sending a command that modifies registers of the managedforwarding element associated with the particular source.
 16. The methodof claim 15, wherein the modified registers specify that the packet isreceived through an ingress port of the managed forwarding element thatconnects to the particular source.
 17. The method of claim 14 furthercomprising examining the received packet to identify the managedforwarding element into which to insert the packet.
 18. The method ofclaim 14, wherein the second network controller performs analysis on thefirst and second sets of messages in order to generate a reportregarding the trace operation.
 19. The method of claim 14, wherein theparticular source is a logical port of the logical forwarding element.20. The method of claim 19, wherein the logical port is associated withan external network to which the logical forwarding element connects,wherein a plurality of different managed forwarding elements areassociated with the logical port, the method further comprisingselecting one of the plurality of different managed forwarding elementsassociated with the logical port as the managed forwarding element intowhich to insert the packet.
 21. The method of claim 19, wherein thelogical port is associated with a remote domain to which the logicalforwarding element connects, wherein a plurality of different managedforwarding elements are associated with the logical port, the methodfurther comprising selecting one of the plurality of different managedforwarding elements associated with the logical port as the managedforwarding element into which to insert the packet.
 22. A machinereadable medium storing a program for execution by at least oneprocessing unit of a first network controller that manages a firstplurality of managed forwarding elements, the program comprising sets ofinstructions for: receiving, from a second network controller, aspecified packet having (i) a particular source on a logical forwardingelement and (ii) an indicator that the packet is for a trace operation;inserting the packet into a managed forwarding element associated withthe particular source; receiving a first set of messages from a firstset of managed forwarding elements in the first plurality of managedforwarding elements that process the packet regarding operationsperformed on the packet; and generating a second set of messages basedon the first set of messages; and sending the second set of messages tothe second network controller in order for the second network controllerto combine the second set of messages with other messages received fromother network controllers that manage other managed forwarding elements.23. The machine readable medium of claim 22, wherein the set ofinstructions for generating the second set of messages comprises a setof instructions for analyzing the first set of messages, wherein thesecond set of messages comprises analysis of the first set of messages.24. The machine readable medium of claim 22, wherein the set ofinstructions for generating the second set of messages comprises a setof instructions for extracting information from the first set ofmessages to include in the second set of messages.
 25. The machinereadable medium of claim 22, wherein the program further comprises setsof instructions for: receiving a third set of messages from a third setof managed forwarding elements in the plurality of managed forwardingelements that process a different packet generated by a third networkcontroller; generating a fourth set of messages by analyzing the thirdset of messages; and sending the fourth set of messages to the thirdnetwork controller in order for the third network controller to combinethe fourth set of messages with other messages received from othernetwork controllers.
 26. The machine readable medium of claim 22,wherein the second network controller generates a report for the traceoperation based on the second set of messages and the other messagesreceived from the network controllers.